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SUMMARY 


The  Analysis  of  IV&V  Data  Study  was  a  one-year  project  undertaken  to  determine 
the  effects  of  Independent  Verification  and  Validation  (IV&V)  on  software  re¬ 
liability,  maintainability,  development  cost,  and  development  productivity. 
Five  large  IV&V  projects  were  selected  for  study.  Information  was  collected 
from  each  of  the  five  projects  concerning  the  development  effort,  the  IV&V 
effort,  and  each  anomaly  reported  by  IV&V.  Current  literature  was  surveyed 
to  obtain  information  relevant  to  software  reliability,  maintainability,  de¬ 
velopment  cost,  and  development  product i vity.  The  IV&V  results  were  analyzed 
in  the-  light  of  findings  from  the  literature,  and  conclusions  were  drawn  and 
recommendations  formulated  for  improving  IV&V  effectiveness. 

The  results  of  the  study  may  be  summarized  as  follows: 

•  General  Results:  1575  anomalies  were  reported  by  the  5  pro¬ 
jects^  07" these,  1023  concerned  software  reliability,  854  con¬ 
cerned  software  maintainability,  and  167  concerned  efficiency, 
usability,  and  other  effects.  Multiple  effects  cause  the  sum  to 
exceed  1575. 

•  Effects  on  Reliability:  The  primary  concern  of  IV&V  is  software 

rel lability.  i 

Each  IV&V  project  reported  an  average  of  150  anomalies  (2.2 
per  thousand  machine  language  instructions)  that  would  have 
affected  program  reliability  and  that  were  considered  im¬ 
portant  enough  to  be  acted  on. 

-  Of  the  three  programs  that  have  recorded  operational  per¬ 
formance,  none  has  required  modification  to  correct  reli¬ 
ability  problems  after  undergoing  IV&V. 

•  Effects  on  Maintainability:  IV&V  effects  on  maintainability 
varied  dramati caTfyfrom  one  project  to  another,  depending  upon 
project  objectives: 

-  Where  the  IV&V  charter  included  maintainability  as  a  con¬ 
cern,  as  many  as  197  anomalies  concerned  solely  with  main¬ 
tainability  were  reported  and  corrected. 

Where  maintainability  concerns  were  deemphasized,  as  few  as 
three  such  anomalies  were  reported. 

•  Effects  on  Development  Cost/Productivitv: 

-  IV&V  cost  averaged  25$  of  development  cost  and  20$  of  total 
acquisition  cost  on  the  projects  surveyed. 

Cost  savings  resulting  from  the  detection  of  reli ability 
anomalies  alone  ranged  from  5$  to  25$  of  development  cost, 
in  some  cases  exceeding  the  cost  of  the  IV&V  effort. 
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The  detection  of  maintainability  and  other  anomalies  had 
additional  cost  benefits  on  the  software  life  cycle. 

Out  of  125  cost/productivity  factors  identified  from  the 
literature,  IV&V  has  no  effect  at  all  on  90,  indicating 
the  limited  overhead  that  IV&V  places  on  the  development 
process. 

Of  the  remaining  35  factors,  IV&V  has  a  positive  effect  on 
27  and  a  negative  effect  on  9  (on  one,  both  a  positive  and 
negative  effect  could  be  seen). 

IV&V  enhances  programmer  productivity  by  decreasing  the 
time  spent  in  false  starts  and  defect  removal. 

The  major  conclusions  of  the  study  were  as  follows: 

•  IV&V  results  depend  on  project  charter  and  directives— IV&V 

finds  the  types  of  problems  it  is  directed  to  find. 

•  IV&V  has  a  significant  effect  on  software  reliability, 

•  IV&V  is  being  underutilized  as  a  tool  for  improving  software 

maintainability. 

•  IV&V  can  pay  for  itself  through  the  detection  of  reliability 

anomalies  alone. 

•  The  cost  benefits  of  ‘IV&V  are  enhanced  by  early  detection  of 
anomalies. 

Major  recommendations  derived  from  these  conclusions  are  as  follows: 

•  To  increase  I V&V * s  effect  on  reliability:  encourage  independ¬ 
ence  of  outlook  and  techniques,  allow  for  early  detection  of 
problems,  and  ensure  that  anomaly  corrections  are  reverified. 

•  To  increase  IV&V's  effect  on  maintainability:  include  a  main¬ 
tainability  evaluation  in  the  IV&V  project's  charter  and  allow 
time  in  the  development  process  for  verification  of  final  pro¬ 
gram  documentation. 

•  To  increase  IV&V  cost  benefits:  begin  IV&V  early  in  the  devel¬ 
opment  process,  require  delivery  of  preliminary  development 
materials,  especially  those  for  requirements  and  design,  and  en¬ 
sure  prompt  action  on  IV&V  findings. 
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This  docent  is  the  technical  report  for  the  Analysis  of  IV&V  Data  Study 
performed  by  Logicon,  Inc.,  under  Contract  F3Q602-80-C-0115  with  the  Rome  Air 
Development  Center  (RADC).  The  work  was  performed  during  the  period  1  April 
1980  to  31  March  1981.'  The  author  of  this  report  is  Jane  k.  Radatz.  Tech¬ 
nical  direction  was  provided  by  Mr.  John  Palaimo,  the  RADC  project  engineer. 
Special  thanks  are  owed  to  Mr.  Donald  Fletcher  (WSM6/RSCS) ,  Captain  John 
Grelck  (BMO/MNNAG),  and  Lt.  Colonel  Thomas  Jarrell  (ASD/YYM),  who  provided 
data  essential  to  the  study.  Significant  contributions  to  the  data  collection 
activity  were  also  made  by  Elaine  Renner,  Norie  Roeder,  and  Joan  Small  of 
Logicon.  Susan  Moy  and  Myra  Chern  provided  expertise  in  the  statistical  anal¬ 
ysis  of  the  IV&V  data.  Marilyn  Fujii,  Edward  Hinton,  Jeffrey  Laub,  and  Dennis 
Meronek  reviewed  this  report  and  provided  valuable  comments  and  suggestions. 
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1. 


INTRODUCTION 


Independent  Verification  and  Validation  ( I V&V )  is  the  systematic  evaluation  of 
a  computer  program  by  an  agency  independent  of  the  developer.  Usually  per¬ 
formed  in  parallel  with  the  software  development  effort,  I V&V  has  as  its  major 
objectives  detecting  development  problems  as  early  as  possible  and  providing 
the  program  office  with  increased  visibility  into  the  development  effort.  The 
basic  tenet  of  this  approach  is  that  the  independence  of  the  IV&V  agency 
provides  a  fresh  viewpoint,  an  objective  attitude,  and  tools  and  techniques 
specifically  designed  for  error  detection. 

Current  trends  in  large-scale  Department  of  Defense  (DoD)  software  procurement 
are  toward  increasing  use  of  I V&V  as  a  managerial  aid.  Air  Force  Regulation 
122-9  requires  I  V&V  for  all  software  that  exercises  direct  control  over  nu¬ 
clear  weapons.  The  Navy's  software  life  cycle  management  guide,  NAVELEXINST 
5200.23,  "strongly  recommends"  full  IV&V  for  all  projects  in  certain  cate¬ 
gories  and  requires  program  managers  to  "keep  in  mind  the  advantages  of  having 
an  IV&V  contractor"  when  planning  a  software  development  project.  Additional 
Air  Force  policy  statements  are  expected  in  the  near  future  making  IV&V  a  part 
of  the  development  of  all  embedded  computer  systems.  This  increasing  use  of 
IV&V  has  resulted  from  growing  concern  for  the  quality  of  computer  software 
and  from  recognition  of  the  serious  impact  of  software  errors  on  critical  and 
costly  systems. 

On  most  projects,  use  of  IV&V  is  at  the  discretion  of  the  program  manager. 
Because  IV&V  can  be  a  significant  cost  factor  in  a  software  development  proj¬ 
ect,  the  decisions  of  whether  and  how  to  use  it  are  major  ones.  Until  now,  no 
quantitative  data  have  been  available  to  help  a  program  manager  make  these  de¬ 
cisions  or  to  provide  baselines  against  which  IV&V  results  could  be  measured. 
The  Analysis  of  IV&V  Data  Study  was  undertaken  to  provide  this  information. 

The  objectives  of  the  study  were  as  follows: 

t  To  determine  the  effects  of  IV&V  on  software  reliability,  main¬ 
tainability,  development  cost,  and  development  productivity 

•  To  formulate  recommendations  for  improving  the  effectiveness  of 
IV&V  on  future  projects 

The  approach  that  was  taken  was  to  obtain  the  results  of  actual  IV&V  projects, 
to  identify  significant  similarities  and  differences  among  them,  and  to  cor¬ 
relate  the  findings  with  results  found  in  the  literature.  Section  2  describes 
the  projects  that  were  used  for  the  study.  Sections  3  through  6  present  re¬ 
sults  of  the  data  analysis.  Conclusions  and  recommendations  are  presented  in 
Section  7.  Details  of  the  technical  approach  are  given  in  Appendixes  A  and  B. 
Appendix  C  identifies  software  features  that  contribute  to  maintainability. 
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2. 


IV&V  PROJECT  CHARACTERISTICS 


The  study  examined  five  large  IV&V  projects.  These  projects  were  selected 
from  35  candidate  projects  based  on  criteria  including  the  size  and  applica¬ 
tion  of  the  system  being  evaluated  and  the  availability  of  data  for  the  IV&V 
study.  Table  1  identifies  the  projects  that  were  selected.  The  following 
paragraphs  describe  these  projects  in  terms  of  the  development  projects  under¬ 
going  IV&V  and  the  characteristics  of  the  IV&V  projects., 

2.1  Development  Project  Characteristics 

The  development  efforts  evaluated  by  the  five  IV&V  projects  involved  a  variety 
of  scientific  applications  and  employed  different  development  techniques. 
Their  basic  characteristics  are  summarized  below. 

2.1.1  Project  1 

Project  1  evaluated  three  interrelated  programs: 

•  Two  command  and  control  programs 

•  A  mathematical  program  invoked  by  the  command  and  control  pro¬ 
grams 

The  command  and  control  programs  each  consisted  of  24,000  assembly  language 
source  lines  and  were  real-time,  interrupt-driven  programs.  One  was  being 
modified;  the  other  was  undergoing  initial  development.  The  mathematical 
program  consisted  of  39,000  FORTRAN  source  lines  and  2,000  assembly  language 
source  lines.  It  was  a  time-critical  batch  program  (that  is.  It  had  to  finish 
executing  within  a  given  amount  of  time)  and  was  undergoing  initial  develop¬ 
ment. 

Development  of  this  system  took  place  over  a  3-1/2-year  period.  Programming 
practices  included  top-down  program  design,  use  of  a  basic  program  support 
Horary,  and  use  of  a  modified  programmer  team. 

2.1.2  Project  2 

Project  2  evaluated  a  major  modification  to  the  three  programs  involved  in 
Project  1.  Program  sizes,  real-time  characteristics,  and  programming  prac¬ 
tices  remained  virtually  the  same. 

2.1.3  Project  3 

Project  3  evaluated  critical  portions  of  a  missile  tracking  and  analysis  sys¬ 
tem.  The  overall  system  contained  1/6,000  source  lines.  Included  in  the 
study  were  IV&V  efforts  focusing  on  two  key  programs: 

•  A  dedicated  operating  system 

•  A  missile  tracking  program 
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The  operating  system  was  a  real-time  program  consisting  of  14,000  assembly 
language  source  lines.  The  missile  tracking  program  was  a  real-time  program 
consisting  of  23,000  FORTRAN  source  lines  and  16,000  assembly  language  source 
lines. 

Development  of  the  system  took  place  over  a  4-year  period.  Programming  prac¬ 
tices  included  the  use  of  both  full  and  modified  programmer  teams. 

2.1.4  Project  4 

Project  4  evaluated  a  real-time  display  system.  Included  in  the  study  were 
IV&V  efforts  focusing  on  programs  totaling  40,000  FORTRAN  source  lines  and 
1,000  assembly  language  source  lines.  The  system  included  both  real-time  and 
nonreal-time  components  and  was  developed  over  a  period  of  2-1/2  years. 

The  Project  4  development  was  the  only  one  of  the  five  that  was  considered  to 
have  used  modern  programming  practices.  Features  of  the  development  effort 
included: 


•  Top-down  design  at  both  the  system  and  program  levels 

•  Use  of  a  program  design  language 

•  Weekly  walk-throughs  from  program  inception 

•  Use  of  a  modified  programmer  team 

t  Use  of  structured  FORTRAN  implemented  with  a  preprocessor 

•  Development  in  "builds" 

0  Use  of  a  fully  automated  program  support  library 
2.1.5  Project  5 

Project  5  evaluated  portions  of  an  avionics  system  totaling  158,000  lines  of 
JOVIAL  and  assembly  language  code.  Included  in  the  study  were  IV&V  efforts 
focusing  on  a  real-time  portion  consisting  of  52,000  assembly  language  source 
lines. 

Development  of  the  program  took  place  over  a  6-year  period.  Programming  prac¬ 
tices  included  top-down  design  at  both  the  system  and  program  levels  and  the 
use  of  a  modified  programmer  team.  A  unique  feature  of  the  Project  5  develop¬ 
ment  effort  was  a  change  in  charter  that  occurred  during  the  coding  and  check¬ 
out  phase.  Rather  than  completing  the  initial  development  as  originally 
planned,  the  developer  was  redirected  to  explore  alternative  implementations 
as  part  of  the  development  effort. 

2.2  IV&V  Project  Characteristics 

The  standard  approach  to  IV&V  entails  five  distinct  activities  proceeding  in 
parallel  with  the  development  effort: 

0  Requirements  Verification:  Evaluation  of  the  program  require¬ 
ments,  as  documented  in  requirement  specifications,  to  ensure 
that  they  are  clear,  complete,  correct,  and  consistent  with  one 
another  and  with  higher  level  specifications 
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t  Design  Verification:  Evaluation  of  the  preliminary  and  detailed 
iesign,  as  documented  in  the  before-code  design  specification, 
to  ensure  that  it  is  a  complete  and  correct  implementation  of 
the  verified  requirements 

•  Code  Verification:  Inspection  of  the  coded  version  of  the  pro¬ 
gram  to  ensure  that  it  is  a  complete,  correct,  and  (sometimes) 
optimal  implementation  of  the  verified  design 

•  Program  Validation/Testing:  Formal  testing  of  the  program  to 
ensure  that  it  satisfies  its  specified  requirements 

•  Documentation  Verification:  Inspection  of  the  requirement  and 
design  specifications,  and  sometimes  the  user  manuals  and  other 
documents,  to  ensure  that  they  accurately  describe  the  program 
as  implemented 

There  was  considerable  deviation  from  this  process  among  the  projects  sur¬ 
veyed.  The  I  V&V  efforts  comprising  Projects  1  and  2  focused  primarily  on  code 
verification  and  testing.  Requirement  specifications  were  for  the  most  part 
accepted  as  valid  and  used  as  the  baseline  against  which  the  code  was  eval¬ 
uated.  Design  materials  were  not  published  until  well  after  code  production 
was'  under  way,  so  were  not  available  for  design  verification.  Analysis  of  the 
requirement  specifications,  design  specification,  and  other  documents  for 
adequacy  as  program  documentation  was  not  within  the  scope  of  the  projects. 

On  Project  3,  the  requirement  specifications  contained  a  significant  amount  of 
design  material.  As  a  result,  requirements  verification  detected  both  re¬ 
quirement  and  design  problems.  The  before-code  design  specification  contained 
only  high-level  design  information,  preventing  a  detailed  design  verification 
activity  using  this  document.  Code  verification  and  testing  took  place  as 
usual,  and  considerable  attention  was  devoted  to  documentation  verification. 

Project  4  addressed  the  development  effort  that  used  modern  programming  prac¬ 
tices.  On  this  project,  I V&V  participants  took  part  In  the  weekly  walk¬ 
throughs  of  the  evolving  requirements  and  design.  Many  problems  were  reported 
In  these  meetings  rather  than  through  the  normal  medium  of  anomaly  reports.  , 
Code  verification  and  testing  were  performed  as  usual.  Documentation  verifi-  ' 
cation  was  deemphasized,  and  most  documentation  problems  were  reported  by 
letter  rather  than  by  anomaly  report. 

Project  5  was  the  only  one  of  the  five  to  include  a  full  design  verification 
step.  This  project  was  also  nonstandard,  however,  in  that  requirements  ver¬ 
ification  was  not  performed  and  in  that,  with  the  redirection  of  the  develop¬ 
ment  effort  to  explore  alternative  implementations,  the  I V&V  effort  conducted 
extensive  testing  in  support  of  this  new  effort. 

Table  2  identifies  the  software  tools  used  on  the  five  I  V&V  projects.  All 
five  projects  used  both  static  analysis  tools,  which  process  or  evaluate  the 
program  without  executing  it,  and  dynamic  analysis  tools,  which  aid  in  program 
testing  by  providing  a  test  environment,  controlling  and  monitoring  program 
execution,  or  modeling  program  behavior. 
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Table  2.  Software  Tools  Used  on  the  IV&V  Projects 


Tool 


Function 


Projects  1,  2 

Tape  Comparison  Program 
Source  Comparison  Program 
Code  Inspection  Aid 

Software  Environment  Simulator 

Data  Base  Analyzer 

Memory  Decode  Program 
Extension  Register  Analyzer 

Real-Time  Analyzer 

Source  Conversion  Program 

Assembler,  Compiler,  Loader 

Memory  Allocation  Program 

Drum  Memory  Dump  Processor 

Program  Structure  Analyzer 

Interpretive  Computer 
Simulation  (ICS)  of  flight 
Computer 

HUE  Simulation  of  flight 
Program 

ICS  of  Ground  Program  Computer 


Identify  differences  between  object  tapes 

Identify  differences  between  source  files 

Generate  global  cross-references  and  an¬ 
notated  source  listings;  identify  certain 
errors 

Simulation  of  flight  computer,  peripheral 
devices,  and  external  environment 

Generate  set/use  information  from  the 
global  cross-reference 

Translate  load  module  to  source  language 

Verify  correct  setting  of  extension 
registers 

Detect  potential  conflicts  caused  by  in¬ 
terrupts  and  job  priorities 

Convert  source  code  to  execute  on  dif¬ 
ferent  computer 

Verify  correct  assembly,  compilation, 
loading 

Provide  variable  name  and  initial  value 
of  each  location  in  temporary  memory 

Format  and  print  selected  portions  of 
drum  memory 

Confirm  correct  compilation  of  program 

Verify  correct  operation  of  flight  pro¬ 
gram  in  its  target  computer 


Evaluate  flight  program  accuracy  and 
correctness 

3 

Verify  correct  operation  of  Cl  program 
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Table  2.  Software  Tools  Used  on  the  IV&V  Projects  (continued) 


Tool 


Function 


Data  base  Preprocessor 
Branch  Analysis  Program 

Project  3 

Global  Cross-Reference  Gen¬ 
erator 

Source  Comparison  Program 
Interrupt  Intercepter 
Software  Monitor 

Console  Monitor 

Time  Mark  Tool 

Real-Time  Driver 
Console  Test  Program 


Format  and  verify  flight  constants  prior 
to  simulation 

Monitor  program  execution  by  recording 
the  number  of  times  each  code  segment  is 
executed  and  each  branch  outcome  occurs 


Generate  cross-reference  of  program 
variables  and  labels 

Identify  differences  between  source  files 

Intercept  and  modify  real-time  interrupts 

Record  the  state  of  the  system  at  pre¬ 
selected  execution  points 

Capture  transient  displays  on  CRT  console 
and  produce  hard  copy  for  later  analysis 

Monitor  and  record  all  program  requests 
made  to  timer;  simulate  additional  re¬ 
quests 

Create  real-time  tasks  and  monitor  task 
dispatching 

Send  selected  discretes  to  flight  control 
console  to  test  hardware  interface 


Macro  Test  Program 


Verify  correct  operation  of  program 
macros 


Data  Capture  Program 
Plot  Board  Driver 

Task  Network  Simulator 

Test  Driver 


Write  incoming  telemetry  data  on  tape 

Send  data  to  plotboards  to  test  hardware 
i nterf ace 

Simulate  a  network  of  real-time  tasks  to 
test  task  communication  and  queueing 

Drive  operating  system  and  monitor  be¬ 
havior 


Automated  Flowcharter 


Generate  flowcharts  from  code 


-14- 


Table  2.  Software  Tools  Used  on  the  I V&V  Projects  (continued) 


Tool 


Function 


Program  Structure  Analyzer 

Time  Code  Translator  Test 
Program 

Branch  Analysis  Program 

Execution  Tracer 

Data  Base  Modifier 
Data  Base  Access  Program 
Sensor  Data  Extractor 

Routine  Interface  Verifier 

Microfiche  Plot  Program 
History  Tape  Converter 

History  Tape  Perturber 
Data  Flow  Analyzer 

Software  Timer 
Address  Locator 


Perform  symbolic  execution  and  path 
analysis 

Test  hardware  clock 


Monitor  program  execution  by  recording 
the  number  of  times  each  code  segment 
is  executed  and  each  branch  outcome 
occurs 

Execute  in  conjunction  with  instrumented 
code  to  trace  execution,  display  inter¬ 
mediate  values,  and  verify  assertions 

Modify  data  base  values  for  program 
testing 

Permit  batch  access  to  data  base  for 
testing  of  individual  routines 

Extract  sensor  data  from  history  tapes 
for  comparison  and  for  use  in  driviny 
plotboards 

Verify  consistency  and  correctness  of 
routine  interfaces 

Generate  microfiche  plots  of  sensor  data 

Convert  history  tapes  generated  by  one 
system  to  format  suitable  for  testing 
another  system 

Modify  contents  of  history  tape  for  pro¬ 
gram  testing 

Perform  analysis  of  interprocedural  data 
flow  to  detect  potential  mishandling  of 
data 

Monitor  execution  time  of  selected 
modules 

Determine  address  of  any  routine  in  pro¬ 
gram  load  module 
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Table  2.  Software  Tools  Used  on  the  I V&V  Projects  (continued) 


Tool 


Function 


Project  4 

Source  Comparison  Program  Identify  differences  between  source  files 


Global  Cross-Reference  Generate  cross-reference  of  program 

Generator  variables  and  labels 


Automated  Flowcharter 


Generate  flowcharts  from  code 


Program  Structure  Analyzer  Perform  symbolic  execution  and  path 

analysis 

File  Index  and  Source  List  Generate  indexed  source  listing 
Generator 


History  Tape  Dump 


Format  and  print  contents  of  program 
history  tape 


History  Tape  Modifier  Modify  history  tape  for  program  testing 

Load  Module  Comparison  Program  Ensure  that  the  load  module  tested  by 

IV&V  matches  the  load  module  certified 
for  operational  use 


Project  5 

Environmental  Simulator  Simulate  electromagnetic  pulse  environ¬ 

ment 


Electronic  Warfare  Simulator 

Core  Image  Comparator 

Core  Image  Reformatter 

Computer  Interface  Program 

Patch  Processor 

HOL  Simulation  of  Program 


Simulate  flight  computer  and  interfacing 
equipment 

Compare  core  image  before  and  after 
simulation  to  detect  memory  destruction 

Reformat  core  image  for  use  on  another 
computer 

Permit  interfacing  of  two  computers 
Apply  patches  to  a  given  core  image 
Evaluate  algorithms  used  in  program 
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3.  GENERAL  RESULTS 

Table  3  identifies  the  types  of  data  collected  from  the  five  IV&V  projects. 
Analysis  of  this  data  was  performed  with  the  aid  of  the  Statistical  Package 
for  the  Social  Sciences  (SPSS),  a  collection  of  statistical  programs  developed 
by  the  University  of  Chicago  (Reference  1).*  Results  specific  to  IV&V's 
effects  on  software  reliability,  maintainability,  and  cost/productivity  are 
presented  in  Sections  4,  5,  and  6,  respectively.  General  results  useful  as 
background  for  these  findings  are  presented  in  this  section. 

3.1  Number  of  Anomalies  Found 

IV&V  results  are  of  two  general  types: 

•  The  detection  of  anomalies 

•  Assurance  of  the  absence  of  anomalies  of  particular  types  in 
given  segments  of  code  or  documentation 

Both  types  of  results  contribute  to  the  determination  of  software  quality. 
The  first  type,  however,  not  only  determines  software  quality  but  affects  it 
as  well  by  bringing  about  the  correction  of  software  faults.  To  evaluate 
IV&V's  effect  on  software  reliability  and  maintainability,  therefore,  the 
study  focused  on  the  anomaly  detection  aspect  of  IV&V,  looking  at  the  number 
and  types  of  anomalies  detected  and  the  impact  and  resolution  of  these 
anomal ies. 

A  total  of  1575  anomalies  were  reported  by  the  five  IV&V  projects.  The  number 
reported  by  each  project  was  as  follows: 

•  Project  1:  249 

•  Project  2:  325 

•  Project  3:  510 

•  Project  4:  175 

•  Project  5:  316 

To  normalize  these  figures,  they  were  compared  with  the  number  of  machine 
language  instructions  generated  by  the  programs  examined.  The  results  are 
shown  in  Figure  1.  The  overall  average  was  4.6  anomalies  per  thousand  machine 
language  instructions.  Project  3  varied  most  dramatically  from  this  average, 
with  13.4  anomalies  per  thousand  machine  instructions. 

Since  different  projects  operated  under  different  charters  regarding  analysis 
of  specifications  and  other  documents,  a  similar  analysis  was  performed  using 
only  the  code  anomalies  found.  A  total  of  802  code  anomalies  were  reported, 
broken  down  as  follows: 


*RV  1.  H.,  et  al..  Statistical  Packaqe  for  the  Social  Sciences.  McGraw  Hill. 
19,5. 
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Table  3.  Data  Collected  From  the  IV&V  Projects 


•  Data  concerning  each  anomaly  reported  by  IV&V 

Location  (specification,  code,  etc.) 

Type  of  problem 

Probable  effects  if  left  uncorrected 

Severity 

Detection  date 

Detection  method 

Resolution 

-  Resolution  date 

•  Data  concerning  each  IV&V  project 

-  Objectives 
Schedule 
Man-loading 

Relationship  with  developer 
Tools  and  techniques  used 
Cost 

•  Data  concerning  each  development  project 

-  Schedule 

-  Man-loading 

-  Development  practices  used 

-  Test  results 

-  Software  operational  performance 
Software  maintenance  requirements 

-  Cost 


Project  1  Project  2  Project  3  Project  4  Project  5  All  Projects 


Figure  1.  Anomalies  Per  Thousand  Machine  Instructions 

■<* 


•  Project  1:  193 

•  Project  2:  205 

•  Project  3:  111 

•  Project  4:  100 

•  Project  5:  143 

A  comparison  of  these  figures  with  the  number  of  machine  language  instruc¬ 
tions  in  each  program  is  shown  in  Figure  2.  These  results  are  surprisingly 
uniform,  averaging  2.2  code  anomalies  per  thousand  machine  instructions.  The 
somewhat  higher  figures  for  Project  5  probably  result  from  the  experimental 
nature  of  the  development  effort,  with  many  different  versions  of  the  program 
being  tried  over  the  course  of  the  development.  It  is  interesting  to  specu¬ 
late  that  the  low  figure  shown  for  Project  4  results  from  its  use  of  modern 
M  programming  practices,  but  the  sample  size  is  too  small  to  support  such  a 

conclusion.  The  overall  average  of  2.2  code  anomalies  per  thousand  machine 
instructions  is  slightly  higher  than  the  2.0  figure  observed  by  Rubey  in  a 
study  of  IV&V  results  performed  in  1975  (Reference  2).* 

3.2  Distribution  of  Anomalies  Among  Development  Materials 

Figure  3  indicates,  for  each  project,  the  number  of  anomalies  found  in: 

•  Requirement  specifications  before  code  delivery 

•  Design  specifications  before  code  delivery 

•  Code 

•  Requirement  specifications  after  code  delivery 

•  Design  specifications  after  code  delivery 

•  User  documentation  and  other  materials 

The  analysis  was  intended  to  determine  where  and  when  development  problems 
were  likely  to  be  found  by  IV&V.  Instead,  it  illustrates  the  high  dependence 
of  IV&V  results  on  IV&V  and  development  project  characteristics. 

The  results  for  Projects  1  and  2  clearly  reflect  their  focus  on  ensuring  com¬ 
pliance  of  the  code  with  the  requirements.  The  only  surprising  aspect  is  the 
significant  number  of  requirement  anomalies  reported  on  Project  2.  The  re¬ 
sults  for  Project  3  reflect  its  attention  to  all  aspects  of  IV&V,  the  unavail¬ 
ability  of  detailed  design  materials  for  a  full  design  verification,  and  its 
strong  emphasis  on  documentation  verification.  Results  for  Project  4  reflect 
the  reporting  of  requirement,  design,  and  documentation  anomalies  in  meetings 
and  letters  rather  than  anomaly  reports.  Project  5  results  show  its  de- 
ernphasis  on  requirement  verification  and  its  performance  of  design  verifica¬ 
tion,  code  verification,  testing,  and  documentation  verification.  The  change 
of  charter  Imposed  on  the  Project  5  development  effort,  making  it  an  ex¬ 
perimental  rather  than  standard  development  project,  makes  it  impossible  to 
determine  whether  the  performance  of  design  verification  would  have  decreased 
the  number  of  anomalies  found  in  the  code. 


*Rubey,  R.  J.,  "Quantitative  Aspects  of  Software  Validation,"  Proceedings  of 
the  International  Conference  on  Reliable  Software,  April  1975,  pp.  245-2517“ 
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Rl :  Requirement  Specifications  —  before  code 
Di:  Design  Specification  --  before  code 


Figure  3.  Development  Materials  In  Which  Anomalies  Were  Found 


Of  all  five  projects,  only  Project  3  reported  a  significant  number  of  anom¬ 
alies  in  the  user  documentation.  The  other  projects  were  not  chartered  to 
evaluate  these  materials,  and  reported  only  those  anomalies  they  happened  to 
detect  in  trying  to  use  the  documents  themselves. 

3.3  Anomaly  Categories 

Table  4  shows  the  number  of  anomalies  reported  in  each  category  for  each 
project  and  for  the  study  as  a  whole.  Notable  results  are  as  follows: 

•  Among  requirement  anomalies,  incorrect  and  incomplete  require¬ 
ments  predominate,  accounting  for  72%  of  all  requirement  anom¬ 
alies. 

•  Among  before-code  design  specification  anomalies,  reported  al¬ 
most  solely  by  Project  5,  anomalies  concerned  with  choice  of 
algorithm/mathematics,  data  definition,  and  data  handling  pre¬ 
dominate. 

•  Among  code  anomalies: 

-  Projects  1,  2,  and  5  had  as  their  most  prevalent  category 
"choice  of  algorithm  or  mathematics,"  a  design-oriented 
category. 

-  All  projects  reported  a  considerable  number  of  data  handling 
problems. 

-  Overall  results  show  that  code  anomalies  fell  into  the  fol¬ 
lowing  categories,  in  decreasing  order  of  frequency: 

o  Choice  of  algorithm  or  mathematics  (30%) 

o  Data  handling  (24%) 

o  Interfaces,  I/O  (11%) 

o  Requirement/design  compliance;  data  definition  (each  7%) 
o  Other  -jode  problems  (6%) 
o  Sequence  of  operations  (6%) 
o  Timing,  interruptibility  (5%) 
o  Presentation,  standards  compliance  (4%) 

•  For  documentation  anomalies  in  the  after-code  design  specifica¬ 
tion,  prevalent  categories  were  incorrectness,  incompleteness, 
and,  for  Project  3,  presentation  and  standards  compliance. 

Correlation  of  these  results  with  development  project  characteristics  led  to 
the  following  ooservations. 

3.3.1  New  Development  vs.  Modification 

The  Project  2  development  represented  a  modification  to  the  Project  1  soft¬ 
ware.  The  considerably  higher  number  of  requirement  anomalies  for  Proj¬ 
ect  2  may  reflect  a  less  rigorous  requirements  definition  activity  on  the 
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Table  4.  Number  of  Anomalies  Reported  in  Each  Category 


Project 


Anomaly  Category 

1 

T~ 

3 

4 

5 

Requirement  Specification  Anomalies 

Rl.  Incorrect  Requirements 

lb 

55 

73 

16 

4 

163 

R2.  Inconsistent  Requirements 

9 

13 

16 

7 

17 

62 

R3.  Incomplete  Requirements 

22 

27 

54 

29 

b 

138 

R4.  Other  Requirement  Problems 

7 

18 

15 

3 

— 

43 

R5.  Presentation;  Standards  Compliance 

2 

3 

4 

1 

-- 

ID 

162 

56 

27 

416 

Total 

SS 

116 

Before-Code  Design  Specification  Anomalies 

Dl.  Requirement  Compliance 

— 

10 

— 

1 

11 

D2.  Choice  of  Algorithm,  Mathematics 

— 

— 

5 

— 

11 

16 

03.  Sequence  of  Operations 

— 

— 

— 

7 

7 

D4.  Data  Definition 

— 

-- 

-- 

— 

19 

19 

DS.  Data  Handling 

-- 

-- 

— 

— 

18 

18 

DO,  Timing,  Interruptibility 

... 

— 

-- 

-- 

0 

0 

D7.  Interfaces,  1/0 

-- 

— 

— 

— 

8 

8 

D8.  Other  Design  Problems 

— 

— 

1 

— 

0 

1 

D9.  Presentation;  Standards  Compliance 

1 

0 

1 

Total 

~0 

~0 

“17 

~U 

“64 

HI 

Code  Anomalies 

Cl.  Requirement,  Design  Compliance 

13 

6 

26 

9 

2 

56 

C 2.  Choice  of  Algorithm,  Mathematics 

79 

69 

11 

17 

4b 

222 

C3.  Sequence  of  Operations 

8 

4 

12 

3 

18 

45 

C4.  Data  Definition 

6 

22 

1 

21 

6 

56 

CS.  Data  Handling 

34 

62 

25 

24 

32 

177 

CC>.  Timing,  Interruptibility 

24 

9 

6 

39 

C7.  Interfaces,  I/O 

CS.  Other  Code  Problems 

23 

21 

9 

16 

10 

79 

3 

6 

21 

3 

13 

46 

C9.  Presentation;  Standards  Compliance 

3 

6 

6 

7 

10 

32 

Total 

m 

TIT 

loo 

ITS 

752 

After-Code  Design  Specification  Anomalies 

PI*  Incorrect  Documentation 

29 

4 

52 

85 

P2.  Inconsistent  Documentation 

-- 

6 

1 

2 

9 

P3.  Incomplete  Documentation 

*» «* 

W>  «• 

57 

4 

10 

71 

P4.  Other  Documentation  Problems 

•»*» 

«* 

11 

3 

1 

15 

PS.  Presentation,  Standards  Compliance 
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— 

55 

-- 

4 

60 

Total 

u 

“0 

759 

72 

o9 

24u 

User  Documentation  Anomalies 

0 

2 

59 

5 

0 

67 

Other  Anomalies 

1 

2 

2 

19 

Anomalies  in  All  Categories 

249 

32S 

510 

175 

316 

1575 
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modification  activity  than  on  the  initial  development.  The  two  projects  had 
approximately  the  same  number  of  code  anomalies,  with  Project  2  having  con¬ 
siderably  more  anomalies  in  the  categories  of  data  definition  and  data  han¬ 
dling,  and  considerably  fewer  in  requirement/design  compliance  and  timing/ 
interruptibil ity. 

3.3.2  Language  Type 

Project  5  addressed  a  program  written  entirely  in  assembly  language.  It  is 
interesting  to  note  that  this  project  reported  the  most  anomalies  of  any 
project  in  category  C3:  "Sequence  of  operations,"  an  area  notoriously  more 
difficult  in  assembly  language  than  in  higher  order  language.  No  other  lan¬ 
guage-related  results  were  observed. 

3.3.3  Modern  Programming  Practices 

The  system  evaluated  by  Project  4  was  developed  using  modern  programming  prac¬ 
tices.  A  possible  correlation  is  that  Project  4  reported  the  fewest  anomalies 
of  any  project  in  category  C3:  "Sequence  of  operations."  This  result  may  be 
attributable  to  the  use  of  program  design  language  and  structured  programming. 
No  other  trends  were  observed  that  could  be  attributed  to  modern  programming 
practices. 


•3.4  Anomaly  Effects 

Figure  4  indicates  the  number  of  anomalies  on  each  project  that  had  the  poten¬ 
tial  to  affect  software  reliability,  maintainability,  efficiency,  and  usabil¬ 
ity.  The  numbers  may  exceed  project  totals  because  of  the  potential  for  mul¬ 
tiple  effects. 

As  with  anomaly  location,  discussed  in  Section  3.2,  anomaly  effects  reflect 
each  project's  charter  and  objectives.  Projects  1  and  2  were  concerned  almost 
solely  with  software  reliability.  As  a  result,  over  90%  of  the  anomalies  re¬ 
ported  on  these  projects  affected  reliability.  The  other  projects  were  con¬ 
cerned  not  only  with  reliability,  but  with  maintainability,  efficiency,  and 
usability  as  well.  Their  results  present  a  more  balanced  picture. 

Project  3  was  unique  in  reporting  considerably  more  maintainability  than  re¬ 
liability  anomalies.  This  was  the  result  of  its  emphasis  on  documentation 
analysis.  The  number  of  reliability  and  maintainability  anomalies  for  Project 
4  were  almost  the  same.  Project  5  reported  slightly  more  reliability  than 
maintainability  anomalies. 

Overall  results  show  that  85%  of  all  anomalies  affected  reliability,  54% 
affected  maintainability,  4%  affected  efficiency,  and  6%  affected  usability. 
On  all  projects,  anomalies  affecting  efficiency  and  usability  accounted  for 
only  a  small  percentage  of  the  total  number  of  anomalies.  Sections  4  and  5, 
which  focus  on  reliability  and  maintainability  anomalies,  therefore  discuss 
nearly  all  of  the  anomalies  reported. 
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Figure  4.  Predicted  Effects  of  the  Anomalies  Reported 


3.5  Anomaly  Severity 

Figure  5  indicates  the  number  of  anomalies  on  each  project  that  h?d  severity 
ratings  High,  Medium,  Low,  and  Unknown.  The  overall  results  for  the  five 
projects  show  that  approximately  a  tenth  of  the  anomalies  received  High 
ratings,  a  fourth  were  rated  Medium,  and  two  thirds  were  rated  Low.  While  the 
precise  meanings  of  these  ratings  vary  from  one  project  to  another,  they  are 
generally  considered  to  have  the  following  interpretation  for  the  types  of 
programs  considered  here: 

•  High:  Threat  to  life  or  property 

•  Medium:  Serious  threat  to  mission  objectives 

•  Low:  Degraded  system  performance  or  non-operational  effect 

The  seriousness  of  these  consequences  indicates  the  significance  of  the  110 
High-  and  404  Medium-severity  anomalies  reported. 

There  is  a  connection  between  the  type  of  anomalies  reported  on  a  project  and 
the  severity  rating  results.  Projects  3,  4,  and  5,  which  reported  a  signifi¬ 
cant  percentage  of  maintainability  anomalies,  show  higher  percentages  of  Low 
ratings  than  Projects  1  and  2,  which  concentrated  on  reliability  problems.  To 
arrive  at  a  more  accurate  comparison,  the  same  analysis  was  performed  using 
only  the  anomalies  concerned  with  program  code.  Figure  6  shows  the  results. 
The  overall  figures  show  that  over  a  tenth  of  all  code  anomalies  received  High 
ratings,  41%  received  Medium  ratings,  and  just  over  half  received  Low  ratings. 
Again,  the  results  varied  significantly  from  one  project  to  another.  Extremes 
were  Project  1,  which  reported  21%  High,  48%  Medium,  and  30%  Low,  and  Project 
4,  with  1%  High,  12%  Medium,  and  87%  Low.  The  other  projects  had  severity 
ratings  closer  to  the  overall  average. 

3.6  Phase  of  Anomaly  Detection 

Figure  7  shows  the  number  of  anomalies  detected  during  each  development  phase. 
The  results  indicate  the  degree  to  which  I V&V  contributed  to  early  detection 
of  anomalies.  Over  half  of  the  anomalies  were  reported  before  the  developer's 
testing  phase.  Project  4  had  the  most  dramatic  results,  with  89%  of  all  anom¬ 
alies  reported  before  development  testing.  Project  5's  results  are  also  im¬ 
pressive,  with  78%  of  all  anomalies  detected  before  the  testing  phase.  Sec¬ 
tions  5  and  6  discuss  the  benefits  of  early  detection. 

3*7  Anomaly  Report  Acceptance 

Figure  8  shows  the  percentage  of  anomaly  reports  on  each  project  that: 

•  Were  accepted  by  the  program  office  as  valid 

•  Were  accepted  with  changes 

•  Were  rejected  by  the  program  office 

•  Were  withdrawn  by  the  l V&V  contractor  or  superseded  by  other 
reports 

•  Had  unknown  acceptance 
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Severity  Ratings  of  Code  Anomalies 


Figure  7.  Anomalies  Found  in  Each  Development  Phase 


The  results  indicate  the  degree  to  which  IV&V  results  were  both  valid  and 
relevant  to  the  software  development  effort.  The  acceptance  rate  on  all 
projects  was  high,  ranging  from  83%  to  98%.  Taking  into  consideration  anomaly 
reports  that  were  accepted  with  changes,  the  acceptance  rate  ranged  from  88% 
to  98%.  Of  the  anomaly  reports  for  which  acceptance  was  known,  93%  were 
accepted,  an  indication  of  the  high  validity  of  IV&V  results. 

3.8  Anomaly  Resolution 

Figure  9  shows  the  percentage  of  anomalies  on  each  project  for  which: 

•  Action  was  taken 

•  Action  was  not  taken 

•  Resolution  was  open  or  unknown  at  the  time  of  the  study 

Anomalies  were  considered  to  have  been  acted  on  if  they  were  fixed  during  the 
project,  fixed  in  a  program  update,  negated  by  an  unrelated  change,  or  dealt 
with  by  a  work-around  solution.  Anomalies  were  considered  not  to  have  been 
acted  on  if  they  were  rejected,  withdrawn,  superseded,  or  accepted  but  left 
unchanged.  Anomalies  were  considered  to  have  open  or  unknown  resolution  if 
resolution  was  deferred  to  a  program  update  that  had  not  taken  place  at  the 
time  of  the  study  or  if  resolution  could  not  be  determined. 

Projects  1  through  4  have  similar  profiles,  indicating  a  very  high  percentage 
of  anomalies  acted  on.  These  results  indicate  that  IV&V  results  were  not  only 
valid  but  were  sufficiently  important  to  require  corrective  action.  The 
atypical  figures  for  Project  5  are  attributable  to  the  experimental  nature  of 
the  development  project.  Anomalies  reported  in  the  different  program  versions 
did  not  necessarily  require  correction.  These  figures  are  not  typical  of  most 
IV&V  projects. 

3.9  Data  Relationships 

Tables  5  and  6  present  the  results  of  statistical  analyses  examining  the  re¬ 
lationships  between  selected  anomaly  characteristics.  The  chi-square  statis¬ 
tical  procedure  was  used.  A  high  chi-square  value  indicates  the  possibility 
that  the  two  variables  are  statistically  related.  The  procedure  assumes  that 
there  is  no  association,  then  computes  the  probability  of  observing  in  re¬ 
peated  samples  a  relationship  less  pronounced  than  that  in  the  current  sample. 
When  this  probability  is  less  than  5%,  there  is  statistical  evidence  that  the 
two  variables  are  related;  when  it  is  less  than  1%,  the  statistical  evidence 
is  even  stronger. 

The  chi-square  statistics  measure  the  degree  of  relationship  between  each  pair 
of  variables.  This  relationship  may  or  may  not  be  one  of  cause  and  effect. 
The  nature  of  the  relationship  may  be  determined  by  further  examination  of  the 
data. 

Table  5  shows  the  relationship  between  anomaly  severity  ratings  and  anomaly 
location,  effects,  and  development  phase  when  detected.  The  results  Indicate 
a  significant  relationship  between  severity  and  these  other  anomaly  character¬ 
istics.  Specific  results  revealed  by  examination  of  the  data  are  as  follows: 
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Project  1  Project  2  Project  3  Project  4  Project  5  All  Projects  Projects  1-4 


Table  5.  Anomaly  Severity  Relationships 
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Table  6.  Anomaly  Resolution  Relationships 
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•  Code  anomalies  are  more  likely  to  be  rated  High  or  Medium  than 
anomalies  in  other  development  materials. 

•  Anomalies  in  the  after-code  design  specification  and  user 
documentation  are  almost  always  rated  Low. 

§  Anomalies  affecting  reliability  are  the  only  type  likely  to  be 
assigned  High  ratings. 

•  Most  anomalies  with  maintainability  as  their  primary  effect  are 
rated  Low. 

•  Anomalies  detected  during  the  requirements  definition  phase  are 
more  likely  to  be  rated  High  than  those  detected  later. 

•  Anomalies  detected  during  the  coding  and  testing  phases  have 
severity  ratings  very  close  to  the  overall  average  of  7.2%  High, 
26.4%  Medium,  66.4%  Low. 

Table  6  shows  the  relationship  between  various  anomaly  characteristics  and 
anomaly  resolution.  Here  again,  significant  relationships  exist.  Discounting 
results  attributable  to  the  atypical  resolution  pattern  of  Project  5,  the  fol¬ 
lowing  results  can  be  observed  from  the  data: 

•  Anomalies  in  all  categories  are  far  more  likely  to  be  acted  on 
than  not. 

•  Anomalies  affecting  maintainability  and  usability,  while  seem¬ 
ingly  less  significant  than  those  affecting  reliability,  have  an 
even  higher  probability  of  being  acted  on  than  reliability 
anomalies;  anomalies  concerned  with  efficiency  have  a  lower 
probability. 

t  Not  surprisingly,  anomalies  with  High  severity  have  the  greatest 
probability  of  being  acted  on;  the  probabilities  for  Medium  and 
Low  anomalies  are  approximately  equal. 

•  Anomalies  detected  during  the  coding  and  checkout  phase  of  de¬ 
velopment  are  the  most  likely  to  be  acted  on. 


4. 


RESULTS  CONCERNING  SOFTWARE  RELIABILITY 


The  primary  concern  of  I V&V  is  software  reliability,  defined  by  Boehm  as  the 
extent  to  which  software  can  be  expected  to  perform  its  intended  functions 
satisfactorily  (Reference  3).*  Included  within  the  scope  of  reliability  are: 

•  Operational  Correctness:  Ensuring  that  the  software  performs 
all  intended  functions  satisfactorily  and  performs  no  unintended 
functions 

•  Operational  Accuracy:  Ensuring  that  mathematical  functions  are 
performed  with  the  required  accuracy/precision 

•  Operational  Security:  Ensuring  that  the  program  is  free  of  un¬ 
authorized  coding  and  incorporates  all  required  measures  to 
prevent  access  to  software  and  data  by  unauthorized  persons 

The  major  difficulty  in  evaluating  I V&V 1 s  effect  on  software  reliability  is 
the  possibility  that  the  developer  may  eventually  have  detected  some  or  all  of 
the  problems  reported  by  the  I V&V  agency  without  the  latter's  help.  The  fact 
that  I V&V  was  the  first  to  find  them  proves  that: 

•  I V&V  is  capable  of  detecting  development  problems. 

•  IV&V  provides  visibility  into  the  development  process. 

•  IV&V  finds  problems  earlier  than  development  testing  and  may 
therefore  prevent  the  schedule  slips  and  cost  overruns  that 
result  from  late  detection. 

It  does  not  necessarily  prove  that  without  the  aid  of  IV&V,  these  problems 
would  have  gone  undetected  into  the  operational  environment. 

The  ideal  experiment  for  evaluating  IV&V  *  s  effect  on  software  reliability 
would  be  to  have  two  groups  of  equally  experienced  and  talented  programmers 
working  in  equivalent  development  environments  develop  the  same  program  using 
the  same  methods  and  tools.  An  IV&V  group  would  be  assigned  to  one  of  the 
development  efforts,  and  the  resulting  programs  would  be  compared  for  **e- 
liability.  If  the  software  that  had  undergone  IV&V  was  more  reliable  than 
that  developed  without  it,  it  could  be  concluded  that  IV&V  did  Indeed  have  a 
positive  effect. 

The  IV&V  study  was  forced  to  take  a  far  more  limited  approach,  consisting  of 
surveying  the  literature  for  relevant  results  and  examining  the  data  from  the 
five  IV&V  projects  in  light  of  these  research  findings.  The  results  of  these 
activities  are  described  in  the  following  paragraphs. 


♦6'oehm,  "6.  W. ,  et  al.,  “Characteristics  of  Software  Quality,"  TRW  Software 
Series  TRW-SS-73-09,  Dec.  1973. 
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4.1 


Relevant  Findings  in  the  Literature 


A  number  of  studies  have  noted  the  effects  of  submitting  a  program  to  two 
or  more  test  and  evaluation  groups  in  succession.  Proceedings  of  a  TRW 
symposium  on  software  development  (Reference  4)*  reported  that  on  a  large 
development  project,  each  successive  phase  of  testing  followed  the  same 
pattern.  Faults  were  found  at  a  high  rate  at  the  beginning  of  the  phase,  then 
at  lower  and  lower  rates  as  the  phase  continued.  When  the  program  was  turned 
over  to  a  new  test  group  for  the  next  testing  phase,  the  detection  rate  jumped 
up  sharply  and  the  pattern  began  anew.  The  report  theorized  that  the  dif¬ 
ferent  techniques  of  each  test  group  resulted  in  the  renewed  fault  detection 
rate. 

Thayer  reported  similar  findings  in  a  study  of  five  large  development  efforts 
(Reference  5).t  He  identified  as  the  cause  of  this  phenomenon  the  expanded 
test  objectives  and  fresh  viewpoint  of  each  successive  test  group.  In  his 
study,  successive  test  groups  sometimes  found  more  faults  than  their  pred¬ 
ecessors  had. 

Two  other  findings  of  the  Thayer  study  are  also  worthy  of  note.  The  first  is 
that  each  test  group  detected  faults  that  should  have  been  detected  in  pre¬ 
vious  test  phases.  That  is,  in  addition  to  those  faults  detected  because  of 
expanded  test  objectives,  each  test  group  detected  faults  within  the  scope  of 
previous  test  efforts,  The  fresh  viewpoint  and  different  test  techniques  of 
each  group  were  considered  to  be  the  factors  here. 

The  second  finding  was  the  tendency  of  each  test  group,  and  in  particular  each 
test  analyst,  to  report  several  faults  of  a  similar  type  over  a  period  of  a 
day  or  two.  Having  detected  a  certain  *ype  of  fault,  the  analyst  made  a 
specific  search  for  that  type  of  problem  in  other  parts  of  the  program. 
Thayer  states  that  this  tendency  can  have  very  positive  effects  on  the  rate 
and  completeness  of  fault  discovery,  especially  if  the  analyst  is  intimately 
familar  with  all  of  the  code  produced  by  a  given  programmer. 

Studies  on  the  effects  of  various  tools  and  techniques  are  also  relevant.  A 
study  by  Shooman  and  Bolsky  (Reference  6)*  found  that  a  large  proportion 
of  program  faults  can  be  detected  by  code  inspection  without  resorting  to 
computer  testing,  A  study  by  Rubey  (Reference  2)  found  that  analysis  methods 
detect  faults  earlier  than  testing  methods  but  that  both  methods  are  needed  to 


»Prdceedii7g$  of  the  TRW  Symposium  on  Reliable,  Cost-Effective,  Secure  Soft¬ 
ware,  March  19?4,  pp.  b.l3-!>.i?. 

tThayer,  T.  A.  et  a)..  Software  Reliability  Study,  RADC-TR- 76-233,  feb.  1976. 

♦Shooman,  M*  l,t  and  Dolsky,  H.  I.,  "Types,  Distribution,  and  Test  Correction 
Times  for  Programming  Errors,"  Procedures  of  the  International  Conference  on 
Reliable  Software.  April  197S,  pp.  34-357.  ~ 


find  all  types  of  faults.  Finally,  Dana  and  Blizzard  (Reference  7)*  indicate 
that  certain  tools  and  techniques  are  most  effective  in  detecting  each  type  of 
fault. 

A  third  set  of  results  concern  the  benefits  of  early  detection  on  program  re¬ 
liability.  Research  reported  by  Finfer  (Reference  8)t  indicates  that: 

•  The  reliability  of  a  system  is  greatly  affected  when  problems  of 
ohe  development  phase  are  allowed  to  go  uncorrected  into  sub¬ 
sequent  phases. 

•  Design  errors  found  in  integration  and  system  testing  have  a 
much  greater  impact  on  reliability  than  if  they  had  been  de¬ 
tected  during  the  design  phase. 

•  "Crash"  development  and  remediation  efforts  generally  result  in 
poor  system  design  and  poor-quality  software. 

The  implications  of  these  findings  for  I V&V  include  the  following: 

9  The  fresh  viewpoint,  independent  objectives,  and  specialized 
tools  and  techniques  offered  by  IV&V  can  be  expected  to  dis¬ 
close  software  faults  not  detected  by  developer  testing. 

•  The  manual  analysis  techniques  used  by  IV&V  can  be  expected  to 
disclose  software  faults  not  detected  by  developer  testing. 

•  IV&V  analysis  and  testing  combined  can  be-  expected  to  detect 
faults  in  all  categories. 

t  The  early  detection  of  problems  provided  by  IV&V  can  provide  the 
time  needed  for  effective  redesign,  thereby  improving  program 
reliability. 

9  The  IV&V  analyst's  intimate  familiarity  with  the  program  under¬ 
going  evaluation  should  make  possible  the  detection  of  whole 
classes  of  related  faults. 

The  last  phenomenon  is  a  recognized  aspect  of  IV&V.  Often  called  the  "clone 
effect,"  it  accounts  for  the  detection  of  numerous  anomalies  on  most  IV&V 
projects  (Reference  9}.* 


*Dana,  *J.  A.,  and  Blizzard,  J.  D.,  The  Development  of  a  Software  Error  Theory 
to  Classify  and  Detect  Software  Error  ,  L'ogicon  Re'port  HR"-/4bl^,  f4ayT974. 

tFinfer,  M.  C.,  Software  Data  Collection  Study,  Volume  III:  Data  Requirements 
for  Productivity  and  Reliability  Studies,  R ADC^Tr  - 7d-5 2 o  1 7Tl I ,  June  ! 976 . 

*Radatz,  J.  W.,  Ramsey,  0.  C.,  and  McKillop,  T.  L.,  NSCCA/PATE  Guidebooks, 
Volume  III,  logicon  Report  R;SED-802Q4-III,  June  1980. 
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4.2  Project  Results 

The  key  questions  addressed  by  the  study  concerning  IV&V's  effects  on  software 
reliability  were  as  follows: 

•  How  many  of  the  anomaly  reports  submitted  by  IV&V  had  an  effect 
on  program  reliability? 

•  In  what  development  materials  were  the  anomalies  located? 

•  What  types  of  problems  did  they  involve? 

•  What  aspects  of  reliability  did  they  affect? 

•  How  severe  were  their  consequences? 

•  When  were  they  found? 

•  What  was  the  operational  reliability  of  the  completed  programs? 
The  following  paragraphs  discuss  these  issues. 

4.2.1  Number  of  Anomaly  Reports  Affecting  Reliability 

Of  the  1575  anomaly  reports  submitted  on  the  IV&V  projects,  1023  were  con¬ 
cerned  with  software  reliability.  Broken  down  by  project,  the  numbers  were  as 
follows:  1 

Project  1:  229 

•  Project  2:  300 

•  Project  3:  183 

•  Project  4:  95 

•  Project  5:  216 

Only  a  subset  of  these  reports  had  an  actual  effect  on  program  reliability, 
namely,  those  that  were  accepted  as  valid  by  the  program  office  and  acted  on 
by  the  developer.  Figures  10  and  11  show  the  percentage  of  anomaly  reports 
that  met  these  criteria. 

Figure  10  Indicates  program  office  acceptance  of  the  anomaly  reports  concerned 
with  reliability.  On  the  average,  89X  were  accepted  as  written,  an  additional 
2%  were  accepted  with  changes,  7%  were  rejected,  IX  were  withdrawn  or  super¬ 
seded,  and  for  IX,  acceptance  was  unknown.  Project  4  had  the  highest  accept¬ 
ance  rate,  with  98X. 

Figure  11  indicates  the  action  taken  on  reliability  anomalies.  For  reasons 
described  in  Section  3.8,  the  resolution  seen  for  Projects  1-4  is  more  typical 
of  IV&V  projects  than  that  shown  for  Project  5.  The  average  for  these  four 
projects  was  79%  acted  on,  14X  not  acted  on,  and  7S  unknown  or  pending. 

There  is  no  way  of  knowing  how  many  of  these  anomalies  would  have  been  de¬ 
tected  by  the  developer  without  IV&V.  The  results  of  the  literature  search 
imply  that  some  at  least  would  not  have  been.  For  purposes  of  the  study,  the 
following  assumption  was  made:  Any  report  that  was  concerned  with  program 
reliability,  accepted  as  valid  by  the  program  office,  and  acted  on  by  the 
developer  represents  an  improvement  in  program  reliability  attributable  to 
IV&V. 
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10.  Acceptance  of  Anomaly  Reports  Concerned  With  Rel 
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There  were  748  such  anomaly  reports.  For  convenience,  the  anomalies  they  de¬ 
scribe  are  hereafter  referred  to  as  "corrected  reliability  anomalies."  The 
breakdown  of  these  anomalies  by  project  was  as  follows: 


Project  1: 

188 

Project  2: 

216 

Project  3: 

139 

Project  4: 

90 

Project  5: 

115 

To  normalize  these  figures,  they  were  compared  with  the  number  of  machine  lan¬ 
guage  instructions  generated  by  the  programs  examined.  The  results  are  shown 
in  Figure  12.  Project  3  had  the  highest  number,  with  3.6  per  thousand  machine 
language  instructions;  Project  4  had  the  lowest,  with  1.2.  On  the  average, 
IV&V  resulted  in  the  correction  of  2.2  reliability  anomalies  per  thousand 
machine  language  instructions. 

4.2.2  Anomaly  Location 

Anomalies  affecting  reliability  could  be  found  in  requirement  specifications, 
before-code  design  specifications,  code,  or  other  materials  such  as  trade 
study  reports.  Figure  13  shows  the  number  of  corrected  reliability  anomalies 
found  in  each  of  these  development  materials.  Overall,  33%  were  found  in 
requirement  specifications,  6%  in  before-code  design  specifications,  61%  in 
code,  and  1%  in  other  materials.  On  Project  5,  the  only  project  to  perform  a 
standard  design  verification,  over  a  fourth  of  the-  corrected  reliability 
anomalies  were  in  the  before-code  design  specification.  The  other  projects 
reported  most  or  all  of  the  anomalies  in  requirement  specifications  and  code. 

4.2.3  Anomaly  Categories 

Table  7  indicates  the  number  of  corrected  reliability  anomalies  found  in  each 
anomaly  category.  Significant  results  are  as  follows: 

•  IV&V  resulted  in  the  correction  of  245  requirement  anomalies 
that  would  have  affected  reliability.  In  38%  of  these  cases, 
the  requirements  were  incorrect;  in  another  28%,  they  were  in¬ 
complete.  In  21%  the  requirements  were  inconsistent;  in  12% 
they  were  ambiguous,  unfeasible,  or  otherwise  unsatisfactory  for 
software  reliability. 

•  IV&V  resulted  in  the  correction  of  448  code  anomalies  that  would 
have  affected  reliability.  Over  a  third  concerned  an  incorrect 
or  unsatisfactory  choice  of  algorithm  or  mathematics  for  the 
program.  Nearly  a  fourth  concerned  incorrect  handliny  of  pro¬ 
gram  data.  An  additional  11%  were  concerned  with  incorrect  in¬ 
terfaces  or  program  input/output. 

•  IV&V  methods  resulted  in  the  detection  of  code  anomalies  in  all 
categories.  It  could  not  be  determined  how  many  of  these 
anomalies  resulted  from  the  "clone  effect." 
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Table  7.  Number  of  Corrected  Reliability  Anomalies  in  Each  Category 


Project 

Anomaly  Category _  1  £  3  4  5  AT T 


Requirement  Specification  Anomalies 

Rl.  Incorrect  Requirements 

12 

47 

32 

3 

— 

94 

R2.  Inconsistent  Requirements 

8 

9 

12 

6 

17 

52 

R3.  Incomplete  Requirements 

16 

22 

21 

9 

2 

70 

R4.  Other  Requirement  Problems 

5 

15 

8 

1 

— 

29 

R5.  Presentation;  Standards  Compliance 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Total 

41 

93 

73 

19 

19 

245 

Before-Code  Design  Specification  Anomalies 

Dl.  Requirement  Compliance 

— 

— 

8 

— 

1 

9 

D2.  Choice  of  Algorithm,  Mathematics 

— 

— 

2 

— 

7 

9 

D3.  Sequence  of  Operations 

— 

— 

— 

— 

5 

5 

04.  Data  Definition 

— 

-- 

— 

— 

1 

1 

D5.  Data  Handling 

— 

— 

— 

— 

14 

14 

D6.  Timing,  Interruptibility 

— 

-- 

— 

— 

— 

*- 

D7.  Interfaces,  I/O 

08.  Other  Design  Problems 

*•** 

2 

2 

D9.  Presentation;  Standards  Compliance 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Total 

-- 

-- 

10 

-- 

30 

40 

Code  Anomalies 

Cl.  Requirement,  Design  Compliance 

5 

3 

14 

5 

1 

28 

C2.  Choice  of  Algorithm,  Mathematics 

71 

46 

7 

15 

24 

163 

C3.  Sequence  of  Operations 

7 

3 

9 

3 

10 

32 

C4.  Data  Definition 

6 

16 

1 

18 

1 

42 

C5.  Data  Handling 

22 

34 

16 

16 

16 

104 

C6.  Timing,  Interruptibility 

18 

6 

— 

— 

2 

26 

C7.  Interfaces,  I/O 

17 

14 

5 

9 

6 

51 

C8.  Other  Code  Problems 

— 

1 

1 

— 

— 

2 

C9.  Presentation;  Standards  Compliance 
Total 

N/A 

146 

m 

IZ5 

*8 

N/A 

66 

N/A 

60 

N/A 

m 

After-Code  Design  Specification  Anomalies 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

User  Documentation  Anomalies 

N/A 

N/A 

N/A 

N/A 

N/A 

N/A 

Other  Anomalies 

1 

— 

3 

5 

6 

15 

Anomalies  in  All  Categories 

188 

216 

139 

90 

115 

748 
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4.2.4  Aspects  of  Reliability  That  Were  Affected 


The  key  aspects  of  software  reliability  are  operational  correctness,  accuracy, 
and  security.  Figure  14  indicates  the  distribution  of  each  project's  cor¬ 
rected  reliability  anomalies  into  these  three  areas.  Totals  may  exceed  the 
number  of  anomalies  reported  because  of  multiple  effects. 

By  far  the  most  prevalent  aspect  was  operational  correctness.  Most  of  the 
anomalies  for  Projects  1,  2,  and  3,  and  all  for  Projects  4  and  5  fell  into 
this  category.  This  preponderance  is  partly  because  the  other  two  aspects  of 
reliability  do  not  apply  to  all  types  of  software.  Accuracy  applies  primarily 
to  programs  that  perform  calculations  for  which  various  degrees  of  accuracy  or 
precision  can  be  achieved.  Anomalies  concerned  with  this  aspect  of  reliabili¬ 
ty  were  reported  for  Projects  1,  2,  and  3.  Security  applies  to  software  that 
can  be  threatened  with  unauthorized  alteration  or  misuse.  This  aspect  was 
limited  to  Projects  1  and  2,  and  accounted  for  only  a  small  percentage  of  the 
anomalies  on  these  projects.  The  greatest  effect  of  IV&V  lies  in  assuring 
that  the  subject  program  operates  as  expected. 

4.2.5  Anomaly  Severity  Ratings 

Figure  15  indicates  the  severity  ratings  assigned  to  the  corrected  reliability 
anomalies.  The  overall  figures  show  that  about  a  tenth  of  the  anomalies 
received  High  ratings,  a  third  were  rated  Medium,  half  were  rated  Low,  and  for 
4%,  the  severity  was  unknown.  Extremes  were  exhibited  by  Project  1,  on  which 
over  two-thirds  had  High  or  Medium  ratings,  and  by  Project  4,  on  which  85%  had 
Low  ratings.  Projects  2,  3,  and  5  fell  closer  to  the  overall  average. 

In  the  context  of  reliability,  these  ratings  have  the  following  general  in¬ 
terpretation:  ’’  , 

•  High:  threat  to  life  or  property 

•  Medium:  serious  threat  to  mission  objectives 

•  Low:  degraded  system  performance 

The  seriousness  of  the  High  and  Medium  impacts  and  the  fact  that  nearly  half 
of  the  anomalies  had  these  ratings  indicates  the  importance  of  these  IV&V 
results. 

4.2.6  Phase  of  Anomaly  Detection 

Figure  16  shows  the  number  of  corrected  reliability  anomalies  detected  during 
each  development  phase.  Nearly  two-thirds  of  the  anomalies  were  reported  be¬ 
fore  development  testing.  On  Project  4,  93%  of  the  anomalies  were  reported 
before  the  testing  phase;  on  Project  3,  81%.  All  projects  except  Project  1 
reported  well  over  half  of  their  corrected  reliability  anomalies  before  de¬ 
velopment  testing. 

The  significance  of  these  findings  lies  in  the  results  reported  by  Finfer: 
Early  detection  of  anomalies  provides  time  for  effective  redesign,  thereby 
improving  program  reliability. 


Figure  14.  Number  of  Anomalies  Affecting  Each  Aspect  of  Reliability 


Severity  Ratings  of  Corrected  Reliabilit, 


Figure  16.  Development  Phase  In  Which  Corrected  Reliability  Anomalies  Were  Detected 


4.2.7  Operational  Performance  of  the  Completed  Programs 

To  assess  the  reliability  of  the  completed  programs,  the  following  questions 
were  considered  by  the  development  project  questionnaire: 

•  How  many  problems  have  been  reported  since  the  program  became 
operational? 

t  If  the  program  has  been  modified,  was  it  due  to  operational 
problems,  requirement  changes,  or  other  reasons? 

The  responses  were  as  follows: 

•  For  the  Project  1  software,  four  problems  were  reported  in 
operational  use.  When  the  software  was  modified,  however,  the 
purpose  was  to  respond  to  requirement  changes  rather  than  to 
correct  any  operational  problems. 

•  For  the  Project  2  software,  no  problems  were  reported  in  opera¬ 
tional  use.  The  software  was  modified  to  respond  to  requirement 
changes. 

•  No  usable  responses  were  given  for  the  Project  3  software; 
records  of  its  performance  were  combined  with  those  of  inter¬ 
facing  programs  and  could  not  be  separated  out. 

t  For  the  Project  4  software,  no  operational  problems  had  been 
reported,  and  the  software  had  not  yet  been  modified. 

§  The  Project  5  software  had  not  yet  been  put  into  operational 
use;  its  operational  performance  and  maintenance  needs  were 
therefore  unknown. 

For  the  three  projects  for  which  data  was  available,  therefore,  none  had  re¬ 
quired  modification  to  correct  a  reliability  problem  encountered  in  the  opera¬ 
tional  environment.  There  is  no  way  to  establish  that  IV&V  was  responsible 
for  this  high  reliability.  It  is  safe  to  say,  however,  that  with  an  average 
of  150  anomaly  reports  per  project  having  a  direct  bearing  on  the  improvement 
of  reliability,  IV&V  made  a  significant  contribution. 
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5. 


RESULTS  CONCERNING  SOFTWARE  MAINTAINABILITY 


The  overall  cost  of  a  software  product  may  be  far  greater  than  the  cost  to 
develop  it.  Figures  cited  by  Miller  (Reference  10),*  Fife  (Reference  11), t 
and  others  indicate  that  software  maintenance  cost— the  cost  of  modifying  a 
program  after  it  has  become  operational— may  account  for  up  to  70%-75%  of  its 
total  life  cycle  cost.  A  1976  paper  by  Prokop  (Reference  12)*  stated  that 
two  out  of  every  three  Navy  programmers  and  computer  systems  analysts  were 
involved  in  maintaining  existing  software.  Nolan  and  Robinson  (Reference 
13)**  have  found  that  all  data  processing  organizations  eventually  reach  a 
stage  in  which  70%  of  the  effort  is  devofed  to  maintenance  activities. 

Modifying  existing  software  is  a  d^ficult,  error-prone  process.  A  study  by 
McGonagle  (Reference  14 )tt  reports  that  19%  of  all  errors  detected  in  the 
software  of  one  organization  resulted  from  unexpected  side  effects  to  other 
changes.  In  Reference  15,**  Boehm  reports  that  even  for  small  modifications 
(1  to  10  instructions),  the  chance  of  a  successful  first  run  is  at  best  50%, 
and  for  larger  changes,  the  success  rate  decreases  steadily  to  about  15%. 
Lehman  (Reference  16)***  states  that  software  tends  to  become  more  and  more 
complex  with  each  change,  making  each  modification  sore  difficult  than  the 
last. 

Increasing  awareness  of  both  the  likelihood  ar.d  the  difficulty  of  software 
maintenance  has  resulted  sn  new  attitudes  toward  software  development. 


WTTir7~C.  R.,  "Software  Maintenance  and  Life  Cycle  Management,"  Software 
Phenomenology-Working  Papers  of  the  Software  Life  Cycle  Management "Wori^' 

sEiriiH  - - — — ■*- - 

tFife,  0.  w» ,  "Software  Management  Standards,"  Software  Phenomenology— 
Working  Papers  of  the  Software  Life  Cycle  H a n a ^menOor kshopj  Airne 
House,  Aug.  1977,  pp."  1)3-510.  ‘ 

♦Prokop,  J.,  Computers  in  the  Navy.  Annapolis,  HO,  Naval  Institute  Press, 
1976. 


♦♦Robinson,  0.  G.,  "Beyond  the  Four  Stages:  What  Next,"  Software  Pheno¬ 
menology— Working  Papers  of  the  Software  life  Cycle  Management  Workshop,' 
Airl ie  House,  Aug.  1977,  pp,  187-2017  **  ^ 

ttHcGonagle,  J.  0.,  A  Study  of  a  Software  Development  Project.  James  P. 
Angerson  and  Co.,  SeptTTsTl. 

♦  ♦Boehm,  8.  W. ,  "Software  and  Its  Impact:  A  Quantitative  Assessment," 
Datamation,  Hay  1973,  pp.  48-59. 

♦♦♦Lehman,  N.  H.,  "Evolution  Dynamics— A  Phenomenology  of  Software  Mainten¬ 
ance,"  Software  Phenomenology— Working  Papers  of  the  Software  Life  Cycle 
Management  Workshop.  Airl i’e "House,  Aug.  l977,  pp.  313-323. 
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Delivered  software  is  no  longer  viewed  as  a  finished  product  not  intended  for 
change.  Instead,  it  is  assumed  that  the  operational  environment  will  be 
dynamic  and  that  software  will  be  required  to  change  along  with  it.  The  re¬ 
sult  is  increasing  emphasis  on  software  that  is  not  only  reliable,  but  main¬ 
tainable  as  wel  1 . 

The  following  approach  was  taken  to  investigate  the  effect  of  IV&V  on  software 
maintainability: 

•  Identify  from  the  literature  software  attributes  that  have  been 
shown  to  contribute  to  maintainability. 

•  Formulate  hypotheses  about  IV&V's  potential  to  affect  these 
attributes. 

•  Analyze  the  results  of  the  five  IV&V  projects  in  light  of  these 
hypotheses. 

The  results  of  these  activities  are  described  in  the  following  paragraphs. 

5.1  Software  Attributes  That  Contribute  to  Maintainability 

V 

Software  maintenance  may  be  performed  to  remove  or  correct  a  software  fault, 
to  add  new  features  or  capabilities,  to  delete  unused  or  undesirable  features, 
or  to  adapt  the  software  to  hardware  changes. (Reference  17).*  Regardless  of 
the  motivation  for  change,  however,  the  maintenance  process  consists  of  under¬ 
standing  the  existing  software,  making  the  needed  changes,  and  revalidating 
the  modified  software  (Reference  18 ) . t  Software  th'at  has  been  designed, 
coded,  and  documented  in  a  way  that  facilitates  these  tasks  is  said  to  be 
"maintainable." 

According  to  Peercy,  the  three  basic  attributes  of  maintainable  software  are: 

•  Understandability:  The  ease  with  which  the  purpose  and  organi¬ 
zation  of  the  software  can  be  grasped 

•  Modifiability:  The  ease  with  which  changes  can  be  incorporated 
once  the  nature  of  the  desired  change  has  been  identified 

•  Testability:  The  extent  to  which  the  software  supports  evalua- 
ti on  "of  its  performance 


*Peercy,  D.  E.,  A  Software  Maintainability  Evaluation  Methodology,"  Proceed- 
inqs  of  the  AIAA  2nd  Computers  in  Aerospace  Conference,  Oct.  1979,  pp.3T5^ 
12b. 

tBoehm,  B.  W.,  "Software  Engineering,"  IEEE  Transactions  on  Computers,  Dec. 
1976,  pp.  1226-1241.  "  ' 
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To  these  three  characteristics  Neil  and  Gold  (Reference  19)*  add: 

#  Portabil ity:  The  ease  with  which  a  software  product  can  be 
transferred  from  one  computer  environment  to  another 

Specific  software  features  contribute  to  each  of  these  attributes.  Features 
that  contribute  to  understandability  include  complete,  accurate  documenta¬ 
tion,  good  traceability  between  code  and  requirements,  and  code  and  design 
that  are  modular,  self-descriptive,  noncomplex,  and  consistent.  Features  that 
contribute  to  modifiability  include  data  structures  designed  to  allow  for  ease 
of  expansion  and  change,  code  and  data  structures  that  minimize  the  side 
effects  of  changes,  and  documentation  that  corresponds  to  the  code  and  is 
modular  in  nature.  Features  that  contribute  to  testability  include  software 
structures  that  isolate  the  effects  of  changes,  program  instrumentation,  and 
complete,  accurate  documentation.  Features  that  contribute  to  portability 
include  device  independence,  use  of  higher  order  language,  and  minimization  of 
interfaces  with  other  systems.  Appendix  C  identifies  more  specifically  a 
variety  of  features  that  contribute  to  software  maintainability. 

5.2  IV&V's  Potential  for  Improving  Maintainability 

Software  maintenance  is  rarely  performed  by  the  original  programmer.  More 
typically,  a  person  unfamiliar  with  the  program  must  study  the  code  and  its 
documentation  until  he  understands  the  program  well  enough  to  make  the  needed 
changes  and  to  devise  test  cases  to  requalify  the  program. 

The  similarity  of  this  process  to  the  IV&V  process  is  striking.  The  IV&V 
analyst  must  study  the  documentation  and  code  until  he  becomes  sufficiently 
familiar  with  the  program  to  follow  the  programmer's  thought  processes,  detect 
logical  flaws,  identify  situations  that  the  programmer  may  have  failed  to  con¬ 
sider,  and  devise  test  cases  to  thoroughly  test  the  program. 

The  objectives  of  the  maintenance  programmer  and  the  IV&V  analyst  differ.  The 
maintenance  programmer  wants  to  modify  the  program,  the  IV&V  analyst  to  eval¬ 
uate  it.  The  similar  preparations  that  both  must  make  to  perform  these  func¬ 
tions,  however,  suggest  that  the  IV&V  analyst  is  In  an  excellent  position  to 
assess  software  maintainability.  The  following  paragraphs  explore  this  hy¬ 
pothesis  by  addressing  each  of  the  four  major  aspects  of  maintainability  and 
the  special  case  in  which  IV&V  is  applied  to  a  maintenance  effort.  A  conclud¬ 
ing  paragraph  discusses  indirect  effects  of  IV&V ' s  assessment  of  software 
rel lability. 

5.2.1  Understandability 

Understandability  is  as  crucial  to  the  IV&V  analyst  as  it  is  to  the  mainte¬ 
nance  programmer.  In  the  analyst's  efforts  to  become  familiar  with  the 
requirement  and  design  specifications,  he  notices  incompleteness,  inconsis- 


Wf,  "6.7  and  Gold,  H.  I.,  Software  Acquisition  Management  Guidebook:  Soft- 
ware  Quality  Assurance.  ESO-TR-77-25&,  Aug.  197>. 


tencies,  inaccuracies,  ambiguities,  unclear  presentation,  and  other  problems 
of  this  type  because  they  hamper  his  own  efforts  to  understand  the  software 
system.  Similarly,  during  the  detailed  code  analysis  that  is  central  to  IV&V, 
such  problems  as  inadequate  or  incorrect  comments,  unstructured  or  unmodular 
code,  complex  constructs,  and  obscure  logic  present  the  IV&V  analyst  with  the 
same  difficulties  they  would  present  the  maintenance  programmer.  Assessing 
understandability  is  therefore  a  natural  part  of  IV&V;  the  findings  can  be 
reported  if  the  program  office  so  chooses. 

5.2.2  Modifiability 

According  to  Peercy,  modifiability  consists  of  understandability  plus  expanda¬ 
bility,  where  expandability  is  the  extent  to  which  a  physical  change  to  in¬ 
formation,  computational  functions,  data  storage,  or  execution  time  can  be 
easily  accomplished.  Software  features  that  contribute  to  expandability 
include  a  reasonable  margin  of  storage  space  and  processing  time,  extra  fields 
in  data  files,  parameterization  of  constants  and  data  structure  sizes,  and 
documentation  that  will  easily  accommodate  change. 

Although  few  if  any  IV&V  projects  have  been  chartered  to  evaluate  software  for 
modifiability,  anomalies  concerning  this  trait  are  often  reported  under  the 
category  "poor  programming  practices,"  and  the  potential  to  expand  this  eval¬ 
uation  certainly  exists.  Data  bases  that  are  being  examined  for  accuracy 
could  be  simultaneously  evaluated  for  expandability.  Code  being  examined  for 
correctness  and  efficiency  could  also  be  evaluated  against  a  set  of  criteria 
known  to  enhance  expandability.  Documentation  being  examined  for  correctness, 
completeness,  and  consistency  could  also  be  evaluated  for  the  type  of  modu¬ 
larity  and  traceability  that  enhance  expandability.  Drawing,  from  a  list  such 
as  that  given  in  Appendix  C,  a  set  of  explicit  modifiability  criteria  could  be 
developed  against  which  the  software  was  to  be  evaluated.  Manual  analyses  of 
code,  data,  and  documentation  would  then  include  these  criteria  along  with 
those  normally  used.  Special  tools  to  evaluate  certain  features  might  also  be 
developed. 


5.2.3  Testability 

Peercy  defines  testability  as  understandability  plus  instrumentation,  where 
instrumentation  is  the  extent  to  which  software  contains  embedded  test  aids  or 
has  been  implemented  to  allow  the  use  of  external  test  aids.  Embedded  aids 

.  might  include  assertions  or  execution  monitoring  statements;  external  test 
aids  might  include  drivers,  monitors,  simulators,  or  te<*t  case  generators. 

An  important  aspect  of  IV&V  requirements  verification  is  evaluation  of  each 
requirement  for  testability.  Considered  in  the  evaluation  are  the  clarity, 
quantifiability,  and  feasibility  of  each  requirement.  While  evaluation  of  the 
design  and  code  for  testability  has  not  traditionally  been  included  in  the 
IV&V  charter,  here  again,  the  potential  exists.  If  the  code  has  been  instru¬ 
mented  with  assertions  indicating  expected  conditions  on  inputs,  outputs, 
program  variables,  or  other  program  aspects,  the  analyst  could  evaluate 
the  completeness  and  quality  of  these  embedded  test  aids.  If  it  does  not  con¬ 
tain  such  instrumentation,  recommendations  could  be  made  for  ways  to  include 
assertions  or  to  accommodate  external  test  aids.  Software  structures  could  be 


-56- 


evaluated  for  their  ability  to  isolate  the  effects  of  change.  These  and  other 
testability  criteria  could  be  developed,  and  deviations  could  be  reported  in 
anomaly  or  other  types  of  reports. 

5.2.4  Portability 

Portability  contributes  to  maintainability  by  reducing  the  need  for  modifica¬ 
tion  when  new  equipment  is  introduced  or  when  the  software  is  transferred  to  a 
new  environment.  Important  aspects  of  portability  are  the  use  of  a  higher 
order  language  and  minimization  of  equipment  dependencies.  Even  when  these 
measures  are  adopted,  however,  portability  is  difficult  to  achieve.  Differ¬ 
ences  in  computer  word  sizes  make  compatibility  between  some  systems  very 
difficult  to  achieve.  Small  segments  of  assembly  language  tend  to  appear  in 
programs  that  are  supposed  to  be  written  in  higher  order  language.  Higher 
order  languages,  despite  their  claims  of  portability,  have  been  adapted  to 
particular  computer  systems. 

From  the  point  of  view  of  IV&V,  portability  is  a  trait  that  can  be  evaluated 
quite  effectively.  Guidebooks  exist  (e.g..  Reference  20)*  which  identify 
higher  order  language  constructs  that  are  truly  hardware  independent.  Cri¬ 
teria  can  be  established  to  evaluate  device  independence.  Evaluation  of  code 
for  portability  could  be  included  as  part  of  the  code  verification  process. 

5.2.5  IV&V  of  Maintenance  Efforts 

The  preceding  sections  were  concerned  with  software  development.  Another 
aspect  of  the  maintainability  issue  is  ensuring  that  software  undergoing 
maintenance  does  not  become  less  maintainable  than  it  was  before. 

In  his  survey  of  software  maintenance  technology  (Reference  21), t  Donahoo 
cites  the  following  four  issues  as  the  major  concerns  in  software  maintenance: 

1)  Lehman's  "Law  of  Increasing  Entropy":  The  complexity  of  a  pro¬ 
gram  tends  to  increase  with  each  modification,  making  mainten¬ 
ance  more  difficult  each  time,  unless  specific  effort  is  applied 
to  stop  this  trend 

2)  The  tendency  for  correction  of  one  problem  to  cause  others  to 
appear 

3)  The  question  of  how  much  of  the  program  to  retest  after  modifi¬ 
cation 


*Georglvi ou,  D.  L.,  Guidelines  for  Programming  in  Portable  Fortran,  Logicon 
Report  No,  DS-R780C9 ,  Gept.  1978.  ~ 


tDonahoo,  J.  D.,  A  Review  of  Software  Maintenance  Technology,  RADC-TK-80-13, 
Feb.  1980. 
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4)  The  tendency  for  program  documentation  not  to  be  updated  to  re¬ 
flect  the  changes  made 

Issues  2  and  3  are  concerned  with  the  reliability  of  the  modified  software. 
These  issues  are  always  addressed  in  the  IV&V  of  maintenance  efforts.  Issue 
4,  concerned  with  continued  maintainability,  is  also  inherent  in  the  IV&V 
process,  through  the  documentation  verification  activity.  While  Issue  1  has 
not  traditionally  been  included  in  the  IV&V  charter,  it  has  been  addressed 
informally  with  the  reporting  of  poor  programming  practices.  By  directing  the 
design  and  code  verification  activities  to  report  all  unwarranted  increases 
in  complexity,  such  as  those  caused  by  artificial  localization  of  changes,  the 
program  office  could  focus  attention  on  this  problem  and  ensure  that  the 
resulting  software  was  not  only  as  reliable,  but  also  as  maintainable  as  it 
was  before. 


5.2.6  Indirect  Effects  on  Maintainability 

IV&V ' s  evaluation  of  software  reliability  has  the  added  effect  of  enhancing 
the  maintainability  of  the  subject  program.  The  improved  reliability  of  the 
software  makes  it  less  likely  to  require  modification  once  in  the  operational 
environment,  and  the  early  detection  of  anomalies  provided  by  IV&V  allows  time 
for  effective  redesign  rather  than  "kluge"  solutions,  which  can  make  the  soft¬ 
ware  overly  complex,  nonmodular,  and  difficult  to  understand. 


Data  from  the  projects  surveyed  provided  answers  to  the  following  questions: 


•  How  many  of  the  anomaly  reports  submitted  by  IV&V  had  a  direct 
effect  on  software  maintainability? 


•  What  development  materials  did  they  involve? 

•  What  types  of  problems  did  they  report? 

•  What  were  their  severity  ratings? 

t  What  aspects  of  maintainability  would  have  been  affected? 

•  What  were  the  indirect  effects  resulting  from  reliability  eval¬ 
uation? 

The  following  paragraphs  discuss  these  issues. 

5.3.1  Number  of  Anomaly  Reports  Affecting  Maintainability 

Of  the  1575  anomalies  reported  on  the  IV&V  projects,  854  were  concerned  with 
software  maintainability.  The  number  of  these  anomalies  on  each  project  was 
as  follows: 


Project  1: 

66 

Project  2: 

135 

Project  3: 

371 

Project  4: 

97 

Project  5: 

185 

Many  of  these  anomalies  had  maintainability  as  one  of  several  effects.  Such 
anomalies  were  usually  reported  for  some  reason  other  than  maintainability  and 
had  maintainability  as  a  secondary  effect.  It  was  instructive,  therefore,  to 
single  out  the  anomalies  that  had  maintainability  as  their  only  effect.  There 
were  347  such  anomalies,  broken  down  as  follows: 


Project  1: 

10 

Project  2: 

9 

Project  3: 

211 

Project  4: 

47 

Project  5: 

70 

This  breakdown  shows  dramatically  the  results  of  different  project  objectives. 
Projects  1  and  2  were  concerned  almost  solely  with  software  reliability.  They 
reported  maintainability  anomalies  only  when  these  might  result  in  reliability 
problems  in  future  program  versions.  Project  3  performed  extensive  documenta¬ 
tion  analysis  aimed  at  detecting  maintainability  problems.  Project  4  was 
discouraged  from  reporting  documentation  problems  in  anomaly  reports.  Project 
5  performed  some  documentation  analysis,  but  had  limited  emphasis  on  maintain¬ 
ability. 

Of  the  854  anomaly  reports  cited  above,  those  that  were  accepted  by  the  pro¬ 
gram  office  and  acted  on  by  the  developer  had  a  direct  effect  on  software 
maintainability.  Figures  17  and  18  indicate  program  office  acceptance  of  the 
two  types  of  maintainability  reports.  Figure  17  shows  that  for  all  maintain¬ 
ability  reports,  an  average  of  90%  were  accepted,  3%  were  rejected,  0%  were 
withdrawn  or  superseded,  and  for  6%  acceptance  was  unknown.  Figure  18  shows 
that  for  anomaly  reports  concerned  solely  with  maintainability,  an  average  of 
94%  were  accepted,  1%  were  rejected,  0%  were  withdrawn  or  superseded,  and  for 
5%  acceptance  was  unknown.  Projects  3  and  4  had  100%  acceptance  of  maintain¬ 
ability-only  reports.  The  projects  that  were  less  concerned  with  maintain¬ 
ability  had  somewhat  lower  rates.  Even  on  these  projects,  however,  acceptance 
was  high  enough  to  indicate  the  overall  validity  of  the  findings. 

Figures  19  and  20  indicate  the  action  taken  on  the  two  types  of  maintainabili¬ 
ty  anomalies.  Figure  19  shows  a  pattern  similar  to  that  observed  for  relia¬ 
bility  anomalies,  namely,  Projects  1-4  exhibiting  similar  profiles  and  Project 
5  being  markedly  different  due  to  the  experimental  nature  of  the  development 
project.  For  Projects  1-4,  an  average  of  80%  of  the  anomalies  were  acted  on, 
9%  were  not,  and  10%  had  resolution  still  open  or  unknown.  Figur*  20  shows 
the  resolution  of  anomalies  that  had  maintainability  as  their  only  effect. 
Projects  2,  3,  and  4  show  high  rates  of  corrective  action;  Projects  1  and  5 
show  lower  rates.  Overall,  79%  of  these  anomalies  were  acted  on,  10%  were 
not,  and  resolution  of  11%  was  still  open  or  unknown  at  the  time  of  the 
study.  Thus,  while  maintainability  anomalies  may  have  had  a  lower  priority 
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Figure  17.  Acceptance  of  Anomaly  Reports  Concerned  With  Maintainability 


Figure  18.  Acceptance  of  Anomaly  Reports  Concerned  Solely  With  Maintainability 


Acted 


than  those  concerned  with  reliability,  they  were  considered  important  enough 
on  three  of  the  projects  to  have  a  high  rate  of  corrective  action. 

The  total  number  of  anomaly  reports  that  were  concerned  with  maintainability, 
accepted  as  valid,  and  acted  on  by  the  developer  was  645.  These  reports  rep¬ 
resent  the  direct  contribution  of  IV&V  to  the  maintainability  of  the  subject 
programs.  The  breakdown  of  these  anomalies,  hereafter  referred  to  as  "cor¬ 
rected  maintainability  anomalies,"  was  as  follows: 


Project  1: 

48 

Project  2: 

108 

Project  3: 

330 

Project  4: 

82 

Project  5: 

77 

Singling  out  the  anomalies  that  affected  maintainability  only,  the  total  was 
276,  broken  down  as  follows: 

•  Project  1:  3 

•  Project  2:  7 

•  Project  3:  207 

•  Project  4:  45 

•  Project  5:  14 

These  anomalies  are  hereafter  referred  to  as  "corrected  maintainability-only 
anomalies."  The  remainder  of  the  analysis  is  concerned  with  these  two  sets  of 
anomalies. 


5.3.2  Anomaly  Location 

Anomalies  affecting  maintainability  may  be  found  in  requirement  specifica¬ 
tions,  before-code  and  after-code  design  specifications,  code,  user  documenta¬ 
tion,  and  other  materials.  Figure  21  shows  the  number  of  corrected  maintain¬ 
ability  anomalies  and  corrected  maintainability-only  anomalies  found  in  each 
of  these  materials. 

Over  half  of  the  corrected  maintainability  anomalies  were  in  requirement 
specifications.  Correction  of  these  anomalies  enhanced  maintainability  by 
making  it  easier  to  understand  the  functions  and  organization  of  the  program 
and  to  devise  new  test  cases.  A  third  of  the  anomalies  were  in  the  before¬ 
code  and  after-code  design  specifications.  Correction  of  these  problems  made 
it  easier  to  grasp  the  program  design  and  to  determine  how  to  make  the  needed 
changes  and  devise  test  cases.  Twelve  per  cent  of  the  anomalies  were  in  the 
code  itself.  Their  correction  resulted  in  code  that  was  traceable  to  require¬ 
ments,  more  efficient,  less  complex,  better  commented,  and  easier  to  under¬ 
stand,  modify,  and  retest. 

5.3.3  Anomaly  Categories 

Table  8  indicates  the  number  of  corrected  maintainability  anomalies  found  in 
each  anomaly  category.  The  left-hand  set  of  figures  applies  to  all  such 
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Table  8.  Number  of  Corrected  Maintainability  Anomalies  in  Each  Category 


Corrected  Maintainability  Corrected  Maintainabil ity- 

_ Anomalies _ _ Only  Anomalies _ 

Project  project 


Anomaly  Category 

1 

2 

3 

T~ 

5 

ATT 

~T~ 

5 

"3 

4 

5 

ITT 

Requirement  Specification  Anomalies 

Rl.  Incorrect  Requirements 

14 

49 

66 

15 

i 

145 

i 

18 

12 

31 

R2.  Inconsistent  Requirements 

9 

9 

13 

7 

17 

55 

— 

-- 

1 

— 

— 

1 

R3.  Incomplete  Requirements 

14 

22 

46 

27 

3 

112 

— 

-- 

23 

14 

— 

37 

R4.  Other  Requirement  Problems 

4 

13 

11 

3 

— 

31 

— 

— 

2 

2 

— 

4 

R5.  Presentation;  Standards  Compliance 

— 

1 

4 

1 

-- 

6 

— 

-- 

1 

1 

... 

2 

Total 

ir 

1* 

U5 

13 

IT 

m 

r 

13 

15 

— 

-75 

Before-Code  Design  Specification  Anomalies 


Dl. 

Requirement  Compliance 

— 

— 

4 

-- 

1 

5 

-- 

-- 

~ 

— 

-X 

— 

02. 

Choice  of  Algorithm,  Mathematics 

x- 

-- 

-- 

— 

7 

7 

— 

— 

— 

— 

-X 

— 

03. 

Sequence  of  Operations 

— 

— 

-- 

— 

5 

5 

— 

— 

— 

— 

XX 

X- 

04. 

Data  Definition 

x- 

— 

— 

-- 

1 

1 

— 

— 

— 

— 

X- 

— 

05. 

Data  Handling 

— 

-- 

— 

-- 

14 

14 

— 

— 

— 

— 

-- 

-- 

06. 

Timing,  Interruptibility 

-- 

-- 

— 

— 

2 

2 

— 

— 

— 

-- 

-- 

— 

07. 

Interfaces,  1/0 

-- 

— 

-x 

— 

— 

— 

— 

-- 

— 

— 

— 

— 

08. 

Cther  Design  Problems 

-- 

-- 

1 

— 

1 

2 

-- 

-- 

1 

X- 

XX 

1 

09. 

Presentation,  Standards  Compliance 

-- 

-- 

-- 

xx 

-- 

-- 

X. 

•  x 

X. 

XX 

•  x 

Total 

Code  Anomalies 

” 

“5 

IT 

1 z 

1 

mm 

■~r 

Cl. 

Requirement,  Design  Compliance 

3 

1 

13 

8 

1 

26 

2 

X. 

X- 

XX 

2 

U. 

Choice  of  Algorithm,  Mathematics 

2 

3 

-- 

— 

— 

5 

— 

1 

— 

— 

1 

C3. 

Sequence  of  Operations 

— 

— 

1 

— 

-x 

1 

— 

— 

1 

-- 

1 

C4. 

Data  Definition 

-- 

3 

1 

3 

— 

7 

•• 

2 

— 

— 

2 

C5. 

Data  Handling 

I 

1 

— 

-- 

1 

3 

i 

1 

-- 

1 

3 

C6. 

Timing,  Interruptibility 

-- 

-- 

x  x 

-- 

— 

- 

XX 

-X 

C7. 

Interfaces,  1/0 

»• 

— 

Xx 

1 

— 

1 

-- 

— 

-X 

X. 

X. 

C8. 

Other  Code  Problems 

1 

2 

10 

1 

U 

25 

a. 

X. 

X. 

-X 

C9. 

Presentation,  Standards  Compliance 

v. 

4 

2 

4 

1 

11 

«• 

2 

2 

4 

1 

9 

Total 

After-Code  Design  Specification  Anomalies 

— 7 

15 

17 

17 

15 

15 

J 

1 

“3 

— $ 

13 

PI. 

Incorrect  Documentation 

— 

m  x 

29 

4 

11 

44 

29 

4 

n 

44 

P2. 

Inconsistent  Documentation 

-- 

xx 

6 

1 

.. 

7 

mm 

•  » 

6 

1 

XX 

1 

P3. 

Incomplete  Documentation 

-- 

»x 

57 

4 

1 

62 

— 

57 

4 

1 

62 

P4. 

Other  Documentation  Problems 

-- 

•  x 

10 

3 

— 

13 

.. 

•  A 

10 

3 

XX 

13 

PS. 

Presentation,  Standards  Compliance 

-• 

56 

— 

56 

h« 

■  X 

56 

.. 

S6 

Total 

— 

— 

13$ 

17 

1? 

T37 

mm 

XSS 

17 

1? 

W 

User  Documentation  Anomalies 

- 

- 

- 

- 

- 

- 

- 

“ 

-- 

.. 

•  m 

Other  Anomalies 

-11 

-H 

-11 

-11 

-11 

-11 

—11 

•  » 

-11 

w* 

Total  Anomalies  in  Alt  Categories 


48  103  BIO  02  77  64b  3 


7  207  45  14  276 
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anomalies;  the  right-hand  set  applies  to  the  anomalies  concerned  with  main¬ 
tainability  alone. 

There  were  349  requirement  specification  anomalies  that  would  have  affected 
maintainability  if  they  had  not  been  corrected.  Most  prevalent  among  these 
were  incorrect  requirements,  which  accounted  for  42%,  and  incomplete  re¬ 
quirements,  which  accounted  for  32%.  Seventy-five  of  these  anomalies  had 
maintainability  as  their  only  effect.  These  anomalies,  reported  only  on 
Projects  3  and  4,  concerned  cases  in  which  the  requirement  specifications  had 
not  been  updated  to  reflect  the  program  as  implemented.  Again,  the  two  most 
prevalent  categories  were  incorrect  and  incomplete  requirements. 

Design  verification  performed  by  Projects  3  and  5  disclosed  36  anomalies  that 
would  have  affected  maintainability  if  they  had  not  been  corrected.  Most 
prevalent  were  errors  in  the  design  specification's  descriptions  of  program 
data  handling.  Only  one  of  the  36  anomalies  had  maintainability  as  its  only 
effect.  This  anomaly  concerned  a  questionable  design  feature  to  be  incor¬ 
porated  in  a  future  update. 

Seventy-nine  code  anomalies  would  have  affected  maintainability  if  they  had 
not  been  corrected.  Most  prevalent  were  cases  in  which  code  was  not  traceable 
to  the  requirements  or  design  and  cases  of  inefficient  or  extraneous  code, 
shown  under  Category  C8.  Eighteen  of  the  code  anomalies  had  maintainability 
as  their  only  effect.  Most  prevalent  here  were  cases  involving  incorrect  and 
incomplete  comments  and  violations  of  development  practices.  Also  included 
were  several  latent  errors,  that  is,  cases  in  whicn  the  program  was  coded  in¬ 
correctly,  but  happened  to  work  satisfactorily  in  the  current  version. 

There  were  182  corrected  maintainability  anomalies  in  after-code  design  spec¬ 
ification,  all  having  maintainability  as  their  only  effect.  Most  prevalent 
were  instances  of  incomplete  documentation.  Problems  with  presentation  of 
information  and  incorrect  documentation  were  also  common. 

b.3.4  Anomaly  Severity 

Severity  ratings  for  all  five  projects  were  based  on  the  impact  an  anomaly 
would  have  on  program  performance.  As  a  result,  anomalies  that  had  as  their 
only  effect  decreased  maintainabil ity  were  almost  always  rated  Low.  The  few 
exceptions  resulted  from  the  fact  that  severity  ratings  were  assigned  to 
entire  anomaly  reports,  rather  than  to  parts  thereof,  and  maintainability 
anomalies  occasionally  received  a  higher  severity  rating  by  association  with 
anomalies  affecting  reliability.  Severity  ratings  for  t  e  anomalies  that  had 
maintainability  as  one  of  several  effects  are  not  meaningful  here  because  the 
ratings  were  generally  assigned  on  the  basis  of  other  effects. 

S.3.5  Maintainability  Attributes  Affected 

Section  $.1  ioentified  as  the  four  basic  attributes  o'.!  maintainable  software 
understandability,  modifiability,  testability,  and  portability.  if.  is  in- 
teiestiny  to  ask  which  of  these  attributes  were  affected  by  the  detection  and 
correction  of  maintainability  anomalies  on  the  five  IViV  projects. 
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The  four  basic  attributes  are  not  mutually  exclusive.  As  described  in  Sec¬ 
tion  5.2,  modifiability  consists  of  understandability  plus  expandability; 
testability  consists  of  understandability  plus  instrumentation.  Any  anomaly 
report  that  contributes  to  understandability  therefore  contributes  to  modifia¬ 
bility  and  testability  as  well. 

To  determine  more  precisely  the  effects  of  the  maintainability  anomalies,  each 
one  was  examined  in  terms  of  its  effect  on  understandability,  expandability, 
instrumentation,  and  portability.  These  traits  are  mutually  exclusive  and 
therefore  more  readily  analyzed. 

The  answer  in  almost  all  cases  was  understandability.  Maintainability  anom¬ 
alies  were  almost  universally  concerned  with: 

•  Documentation  that  did  not  describe  program  requirements  or  de¬ 
sign  completely  or  accurately 

•  Code  characteristics  that  would  make  it  difficult  for  a  main¬ 
tenance  programmer  to  grasp  program  logic  or  to  trace  the  logic 
to  requirements  or  design 

Only  10  anomalies  had  an  effect  other  than  understandability.  These  anom¬ 
alies,  concerned  with  poor  programming  practices  or  latent  errors,  affected 
expandability.  They  would  make  changes  more  difficult  than  necessary  or  make 
it  possible  for  program  changes  to  have  unexpected  side  effects. 

The  predominant  effect  of  I V&V  on  software  maintainability  was  therefore  in¬ 
creased  program  understandability,  and  therefore  enhanced  modifiability  and 
testability  as  well.  The  analysis  showed  that  I  V&V  can  be  an  effective  tool 
for  increasing  software  maintainability. 

5.3.6  Indirect  Effects  of  Reliability  Evaluation 

Section  4  described  748  reliability  anomalies  corrected  as  a  result  of  I  V&V. 
Correction  of  these  anomalies  improved  not  only  program  reliability,  but  also 
program  maintainability.  By  detecting  the  748  problems  before  the  software 
went  into  the  operational  environment,  I  V&V  helped  to  prevent  the  need  for 
corrective  maintenance  of  the  software.  By  detecting  447  of  the  problems  be¬ 
fore  development  testing,  I  V&V  allowed  the  developer  extra  time  for  anomaly 
correction,  reducing  the  possibility  of  poorly  designed  corrections  that  could 
hinder  maintenance  efforts. 
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The  four  basic  attributes  are  not  mutually  exclusive.  As  described  in  Sec¬ 
tion  5.2,  modifiability  consists  of  understandability  plus  expandability; 
testability  consists  of  understandability  plus  instrumentation.  Any  anomaly 
report  that  contributes  to  understandability  therefore  contributes  to  modifia¬ 
bility  and  testability  as  well. 

To  determine  more  precisely  the  effects  of  the  maintainability  anomalies,  each 
one  was  examined  in  terms  of  its  effect  on  understandability,  expandability, 
instrumentation,  and  portability.  These  traits  are  mutually  exclusive  and 
therefore  more  readily  analyzed. 

The  answer  in  almost  all  cases  was  understandability.  Maintainability  anom¬ 
alies  were  a. most  universally  concerned  with: 

•  Documentation  that  did  not  describe  program  requirements  or  de¬ 
sign  completely  or  accurately 

•  Code  characteristics  that  would  make  it  difficult  for  a  main¬ 
tenance  programmer  to  grasp  program  logic  or  to  trace  the  logic 
to  requirements  or  design 

Only  10  anomalies  had  an  effect  other  than  understandability.  These  anom¬ 
alies,  concerned  with  poor  programming  practices  or  latent  errors,  affected 
expandabil ity.  They  would  make  changes  more  difficult  than  necessary  or  make 
it  possible  for  program  changes  to  have  unexpected  side  effects. 

The  predominant,  effect  of  I V&V  on  software  maintainability  was  therefore  in¬ 
creased  program  understandability,  and  therefore  enhanced  modifiability  and 
testability  as  well.  The  analysis  showed  that  I  V&V  can  be  an  effective  tool 
for  increasing  software  maintainability. 

5.3.6  Indirect  Effects  of  Reliability  Evaluation 

Section  4  described  748  reliability  anomalies  corrected  as  a  result  of  IV&V. 
Correction  of  these  anomalies  improved  not  only  program  reliability,  but  aHT) 
program  maintainability.  By  detecting  the  748  problems  before  the  software 
went  into  the  operational  environment,  I  V&V  helped  to  prevent  the  need  f6r 
corrective  maintenance  of  the  software.  By  detecting  447  of  the  problems  be¬ 
fore  development  testing,  IV&V  allowed  the  developer  extra  time  for  anomaly 
correction,  reducing  the  possibility  of  poorly  designed  corrections  that  could 
hinder  maintenance  efforts. 


The  four  basic  attributes  are  not  mutually  exclusive.  As  described  in  Sec¬ 
tion  5.2,  modifiability  consists  of  understandabi 1 ity  plus  expandability; 
testability  consists  of  understandabi 1 ity  plus  instrumentation.  Any  anomaly 
report  that  contributes  to  understandabi 1 ity  therefore  contributes  to  modifia¬ 
bility  and  testability  as  well. 

To  determine  more  precisely  the  effects  of  the  maintainability  anomalies,  each 
one  was  examined  in  terms  of  its  effect  on  understandabil ity,  expandability, 
instrumentation,  and  portability.  These  traits  are  mutually  exclusive  and 
therefore  more  readily  analyzed. 

The  answer  in  almost  all  cases  was  understandabi 1 ity.  Maintainability  anom¬ 
alies  were  almost  universally  concerned  with: 

•  Documentation  that  did  not  describe  program  requirements  or  de¬ 
sign  completely  or  accurately 

t  Code  characteristics  that  would  make  it  difficult  for  a  main¬ 
tenance  programmer  to  grasp  program  logic  or  to  trace  the  logic 
to  requirements  or  design 

Only  10  anomalies  had  an  effect  other  than  understandabi 1 ity.  These  anom¬ 
alies,  concerned  with  poor  programming  practices  or  latent  errors,  affected 
expandability.  They  would  make  changes  more  difficult  than  necessary  or  make 
it  possible  for  program  changes  to  have  unexpected  side  effects. 

The  predominant  effect  cf  IV&V  on  software  maintainability  was  therefore  in¬ 
creased  program  understandabil ity,  and  therefore  enhanced  modifiability  and 
testability  as  well.  The  analysis  showed  that  IV&V  can  be  an  effective  tool 
for  increasing  software  maintainability. 

5.3.6  Indirect  Effects  of  Reliability  Evaluation 

\ 

Section  4  described  748  reliability  anomalies  corrected  as  a  result  of  IV&V. 
Correction  of  these  anomalies  improved  not  only  program  reliability,  but  als'b 
program  maintainability.  By  detecting  the  748  problems  before  the  software 
went  into  the  operational  environment,  IV&V  helped  to  prevent  the  need  fbr 
corrective  maintenance  of  the  software.  By  detecting  447  of  the  problems  be¬ 
fore  development  testing,  IV&V  allowed  the  developer  extra  time  for  anomaly 
correction,  reducing  the  possibility  of  poorly  designed  corrections  that  could 
hinder  maintenance  efforts. 
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t  Hypothesize  for  the  selected  factors  which  ones  would  be  af¬ 
fected  positively  and  which  negatively  by  IV&V 

•  Tie  in  study  results  to  test  the  hypotheses  with  actual  data 
and  to  quantify  results  where  possible 

The  results  of  the  first  three  steps  are  given  in  Table  9.  The  following 
paragraphs  discuss  these  results. 

6.1  Factors  Affecting  Development  Cost/Productivity 

Table  9  identifies  125  cost/productivity  factors  described  in  the  literature. 
They  have  been  divided  into  11  basic  categories: 

•  The  nature  of  the  software  to  be  developed 

•  Special  requirements  imposed  on  the  software 

t  The  quality  and  stability  of  the  requirements 

•  The  quality  and  stability  of  the  design 

•  The  quality  and  stability  of  the  code 

•  Personnel  and  organization 

•  Development  methodology  used 

•  Development  facilities  available 

•  Project  management 

•  Amount  of  nonproductive  activity  performed 

•  Cost  factors  unrelated  to  productivity 

The  left-most  column  identifies  one  or  more  references  in  which  each  factor 
was  found.  When  more  than  two  references  cited  a  given  factor,  only  the  first 
few  encountered  were  included  in  the  table. 

6.2  Factors  Affected  b.y  IV&V 

The  right-hand  columns  of  Table  9  present  the  study's  hypotheses  as  to  the 
potential  effect  of  IV&V  on  each  of  the  cost/productivity  factors.  The  pos¬ 
sible  responses  are: 

•  "Positive,"  meaning  that  IV&V  has  the  potential  to  increase 

programmer  productivity  or  decrease  cost  with  respect  to  the 
factor 

t  "Negative,"  meaning  that  IV&V  has  the  potential  to  decrease 

programmer  productivity  or  increase  cost  with  respect  to  the 
factor 

§  "None,"  meaning  that  little  or  no  effect  could  be  envisioned 
under  normal  circumstances 

The  most  striking  finding  is  that  for  90  of  the  125  factors,  IV&V  was  consid¬ 
ered  to  have  no  effect  at  all.  This  finding  shows  that  the  benefits  of  IV&V 
can  be  obtained  without  placing  a  significant  amount  of  overhead  on  the  devel¬ 
opment  process.  Ruled  out  were  all  factors  under  "the  nature  of  the  software 
to  be  developed"  and  most  factors  under  "special  requirements  imposed  on  the 
software,"  “personnel  and  organization,"  "development  facilities,"  and  "costs 
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Table  9.  Factors  Affecting  Development  Cost/Productivity 


Potential  Effects  of  I V&V 


References 

Factor  Positive 

Negative  None 

A.  The  Nature  of  the  Software  to  be  Developed 

23,  24 

A-l.  Type  of  application 

X 

23,  25 

A-2.  Degree  of  innovation  required 

X 

25,  26 

A-3.  Size  of  the  software 

X 

8,  27 

A-4.  Data  base  size/complexity 

X 

25 

A-5.  Number  and  complexity  of  1/0  formats 

X 

23 

A-6.  Data  management  techniques  to  be  used 

X 

23 

A-7.  Multiple-site  installation 

X 

15,  25 

A-8.  Extent  of  decentralization  and  number 
of  interfaces 

X 

23,  27 

A-9.  Real-time  requirements 

X 

28 

A-1Q,  Reimplementation  of  existing  software 

X 

25 

A-ll.  Number  of  other  components  and  sub- 

X 

systems  being  developed  concurrently 
as  part  of  the  system 


B.  Special  Requirements  Imposed  on  the  Software 


23,  26 

B-l.  Quality  requirements 

•  Reliability  requirements 

X 

•  Maintainability  requirements 

X 

•  Efficiency  requirements 

X 

•  Integrity  requirements 

X 

•  Usability  requirements 

X 

•  Testability  requirements 

X 

•  Portability  requirements 

X 

•  Reusability  requirements 

X 

•  Interoperability  requirements 

X 

•  Transportability  requirements 

X 

23,  25 

B-2.  Requirements  for  special  displays  and 

X 

interfaces  with  special  equipment 

23 

8-3.  Testing  requirements 

X 

8,  27 

B-4.  Documentation  requirements 

X 

27 

8-5.  Percentage  of  code  to  be  delivered 

X 
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Table  9.  Factors  Affecting  Development  Cost/Productivity  (continued) 


Potential  Effects  of  I V&V 


References 

Factor 

Positive  Negative  None 

25,  27 

B-6.  Overall  constraints  on  program  design 

X 

23,  27 

8-7.  Constraints  on  program  size 

X 

23,  27 

B-8.  Constraints  on  program  speed 

X 

8 

8-9.  Requirement  to  install  the  system  at 
a  site  other  than  the  development  site 

X 

C. 

Qualify  and  Stability  of  the  Requirements 

27 

C-l .  Customer  experience: 

o  With  the  application 
o  With  data  processing 

X 

X 

23,  27 

C-2.  User  participation  in  requirements 
definition 

X 

25,  27 

C-3.  Programmer  participation  in  require¬ 
ments  definition 

X 

23 

C-4,  Effectiveness  of  communication  among 
user,  customer,  developer,  maintainer 

X 

a,  23, 

26,  29 

C-5.  Completeness,  accuracy,  and  clarity 
of  requirement  specifications 

X 

23,  30 

C-6.  Level  of  change  in  requirements 
during  development 

X  X 

23,  30 

C-7.  Phase  in  which  requirement  changes 
oecur 

X 

0. 

Quality  and  Stability  of  the  Design 

23 

0-1.  Accuracy  of  translation  from  require¬ 
ments  to  design 

X 

3,  23 

U-2.  Quality  of  the  design 

X 

15,  23 

0-3.  Amount  of  changes  to  the  design 

X 

3,  27,  30 

0-4.  Timing  of  design  changes 

X 
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Table  9.  Factors  Affecting  Development  Cost/Productivity  (continued) 


Potential  Effects  of  I V&V 

References  _ Factor _  Positive  Negative  None 

E.  Quality  and  Stability  of  the  Code 

23  E-l.  Accuracy  of  translation  from  design  to  X 

code 

8,  23  E-2.  Quality  of  the  coded  program  X 

15,  23  E-3.  Amount  of  code  changes  needed  X 

8,  27,  30  E-4.  Timing  of  code  changes  X 


F.  Personnel  and  Organization 

F-l.  Development  team  experience 


27,  28, 

31 

•  Witn  similar  applications 

X 

23,  27 

•  With  the  computer  hardware 

X 

27 

•  With  the  language  to  be  used 

X 

15,  26 

F-2. 

Programmer  ability 

X 

25,  27 

F-3. 

Programmer  participation  in  require¬ 
ments  definition 

X 

25,  26, 

29  F-4, 

Amount  of  training  required 

X 

28,  29 

F-5. 

Development  team  organization  and  size 

X 

8,  25 

F-G. 

Development  team  stability 

X 

31 

F-7. 

Development  team  morale 

X 

29 

F-3. 

Development  team  cooperation 

X 

15 

F-9. 

Development  team  objectives 

X 

26 

P-10. 

Appropriateness  of  man-loading 

X 

31 

F-ll. 

Availability  of  personnel  when  needed 

X 

23,  26 

F-12. 

Percentage  of  support  personnel 

X 

6.  Development  Methodology 

23,  24, 

26  6-1. 

Programming  language  used 

X 

23,  20 

6-2. 

Use  of  modern  programing  practices 
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Table  9.  Factors  Affecting  Development  Cost/Productivity  (continued) 


Potential  Effects  of  IV&V 


References 

Factor 

Positive 

Negative  None 

27,  32 

•  Top-down  development 

X 

30 

•  Program  design  language 

X 

23 

i  HIPO  diagrams 

X 

15,  27 

•  Structured  programming 

X 

18,  23 

•  Programming  support  library 

X 

23,  27 

•  Chief  programmer  team 

X 

27,  33 

•  Design  and  code  inspection 

X 

33 

•  Unit  development  folders 

X 

G-3. 

Compliance  with  well-defined 
standards 

X 

8 

G-4. 

Quality  assurance  practices  followed 

X 

33,  34 

G-5. 

Configuration  management  of  the  soft¬ 
ware  and  documentation 

X 

15,  33 

G-6. 

Quality  of  test  plan 

X 

23 

G-7. 

Avoidance  of  hands-on  batch  testing 

X 

31 

G-8. 

Debugging  style 

X 

8,  25 

G-9. 

Error  reporting  and  correcting  pro¬ 
cedures 

X 

X 

23 

15,  23 


Developient  Facilities 


H-l.  Availability  of  computer  hardware 


•  Late  selection  of  target  computer 

•  Concurrent  (level opment  of  hardware 
and  software 


X 

X 


25 

H-2.  Suitability  of  target  computer 

23 

•  Sufficient  speed 

X 

15,  23 

•  Sufficient  memory  size 

X 

23 

H-3.  Development  and  target  computers 
differ 

X 

25,  27 

23 

H-4.  Ease  of  access  to  computer  facilities 

•  Use  of  operational  site  for  devel¬ 

X 

23 

opment 

•  Use  of  another  organization's 

X 

25 

development  facilities 
•  Need  to  share  computing  iacilities 

X 

8 

•  Proximity  of  computing  facilities 

X 
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Table  9.  Factors  Affecting  Development  Cost/Productivity  (continued) 


Potential  Effects  of  IV&V 


References 

Factor 

Positive  Negative 

None 

8 

•  Proximity  of  computing  facilities 

X 

H-5.  Complexity  of  computer  facilities 

23 

25 

•  Number  of  different  development 
locations 

•  Number  of  different  computers 

X 

X 

H-6.  Computer  response  time 

15,  28 

8 

25 

25 

•  Mode  of  operation:  batch  vs. 
on-line 

•  Computer  throughput  rate 

•  Time  lost  to  maintenance 

•  Probability  of  computer  overload 

X 

X 

X 

X 

29,  31 

H-7.  Computer  system  reliability 

X 

25,  29 

H-8,  Computer  sytem  usability 

X 

H-9.  Support  software  and  development 
tool s : 

23,  25 

29,  31 

•  Availability  when  needed 

•  Quality 

X 

;X 

29 

H-1Q.  Adequacy  of  technical  reference 
materials 

X 

29 

M-ll.  Suitability,  comfort  of  work 
envl  ronment 

X 

25,  29 

H-12.  Cooperation  and  responsiveness  of 
support  services 

X 

27 

H-13.  Classified  security  environment 

X 

25 

H-14.  Availability  of  data  for  the  data 
base 

X 

1.  Amount  of  Non-Productive  Activity  Performed 

15,  25 

1-1.  Travel 

X 

1-2.  Meetings,  Interfaces 

24,  26 

27 

25 

•  Internal 

•  With  customer 

•  With  other  agencies 

X 

X 

X 
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Table  9.  Factors  Affecting  Development  Cost/Productivity  (continued) 


Potential 

Effects  of 

IV&V 

References 

Factor 

Positive 

Negative 

None 

30 

1-3. 

Documentation  preparation 

X 

8,22 

1-4. 

Training 

X 

24 

1-5. 

Company  business 

X 

22,  24 

1-6. 

Paperwork 

X 

24,  25 

1-7. 

Delays  for  needed  materials,  compon¬ 
ents,  concurrence,  etc. 

X 

30,  8,  6 

1-8. 

False  starts i  need  for  development 

X 

30,  35,  36 

1-9. 

Software  defect  removal 

X 

0.  Project  Management 

8 

J-l. 

Type  of  contract 

X 

26,  29 

0-2. 

Feasibility  of  schedule 

X 

8,  26 

0-3. 

Allocation  of  resources  to  each  phase 

X 

23,  26 

0-4. 

Completion  of  activities  within  their 
allotted  phase 

X 

23,  34 

0-5. 

Effectiveness  of  cost  monitoring 

X 

24,  29,  31 

0-6. 

Effectivenss  of  progress  monitoring 

X 

24,  29 

0-7. 

Effectiveness  of  personnel  management 

X 

8,  34 

0-3. 

Adequacy  of  formal  reviews  and  audits 

X 

K.  Costs 

Unrelated  to  Productivity 

K-l. 

Computer  hardware 

X 

23,  25 

K-2. 

Secondary  resources  (computer  time, 
documentation  reproduction,  travel, 
etc.) 

X 

25 

K-3. 

Equipment,  office  space 

X 

25 

K-4. 

Simulation  and  test  facilities 

X 

25 

K-5. 

Special  security-related  equipment 

X 

unrelated  to  productivity."  For  these  factors,  the  presence  of  an  IV&V  agency 
either  made  no  difference  at  all,  or  could  make  a  difference  only  in  unlikely 
circumstances. 

Of  the  remaining  factors,  IV&V  was  considered  to  have  a  potentially  positive 
effect  on  27  and  a  potentially  negative  effect  on  9.  On  one  factoi — error 
reporting  and  correcting  procedures--both  a  positive  and  negative  effect 
could  be  foreseen.  The  following  paragraphs  discuss  these  effects. 

6.2.1  Positive  Effects  of  IV&V 

The  study  hypothesized  a  positive  effect  on  27  of  the  cost/productivity  fac¬ 
tors.  These  factors  and  IV&V's  effects  on  them  are  discussed  below.  Related 
topics  have  been  discussed  together  for  brevity. 

6. 2. 1.1  Quality  and  Stability  of  Requirements:  In  Reference  8,  Finfer 
states  that  the  stability  and  quality  of  requirement  specifications  may  be 
the  key  factor  in  programmer  productivity.  Supporting  this  position  is  Doty's 
software  estimation  guide  (Reference  23),  which  states  that: 

«  Vague  operational  requirements  can  be  expected  to  decrease  pro¬ 
ductivity  by  35%  on  command  and  control  applications  and  50% 
on  scientific  applications. 

t  The  effects  of  changing  requirements  can  be  as  high  as  a  95% 
decrease  in  programmer  productivity. 

Equally  dramatic  are  figures  cited  by  Wolverton  (Reference  30),  which  indi¬ 
cate  that  a  requirement  defect  not  corrected  during  the  requirements  defini¬ 
tion  phase  is: 


•  2-1/2  times  more  costly  to  correct  during  design 

•  5  times  more  costly  to  correct  during  coding  and  checkout 

•  36  times  more  costly  to  correct  during  test  and  integration 

Improving  the  quality  of  requirement  specifications  is  one  of  the  key  objec¬ 
tives  of  IVSV.  Requirements  verification  focuses  on  the  clarity,  complete¬ 
ness,  correctness,  and  consistency  of  requirements.  It  can  point  out  omis¬ 
sions  i  Identify  unfeasible,  questionable,  and  ambiguous  requirements;  detect 
errors;  point  out  inconsistencies;  and  identify  problems  in  the  way  that  the 
requirements  have  been  documented.  The  increased  visibility  provided  by  this 
analysis  can  improve  communication  among  the  user,  customer,  and  developer; 
help  the  user  to  define  hi s  requirements  more  precisely  so  that  later  changes 
will  be  unnecessary;  and  result  in  vastly  improved  requirement  specifications. 
By  performing  this  verification  in  parallel  with  the  requirements  definition 
process,  IV&V  can  ensure  that  requirement  changes  are  made  early,  preventing 
the  major  cost  impacts  associated  with  later  changes. 

6.2. 1.2  Quality  and  Stability  of  the  Design:  According  to  Doty  (Reference 
23): 

•  60%  of  all  errors  discovered  in  testing  are  caused  by  faulty 
design. 
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•  These  errors  can  result  in  cost  increases  of  up  to  100%. 


ffim  III'  importance  of  early  detection,  Wolverton  (Reference  30)  cites 
figures  stating  that  a  design  change  costs,  on  the  average,  $977  to  correct 
during  code  and  checkout  and  $7136  during  test  and  integration  (1975  figures). 


The  design  verification  performed  as  part  of  I V&V  can  have  a  major  impact  on 
design  quality  and  stability.  It  ensures  that  all  requirements  have  been 
properly  translated  into  design  and  that  all  functions  included  in  the  design 
are  traceable  to  approved  requirements.  It  evaluates  the  choice  of  algo¬ 
rithms,  the  design  logic,  the  data  definitions,  and  all  aspects  of  the  design. 
By  performing  this  verification  during  the  design  phase,  I  V&V  can  detect 
design  errors  before  they  are  implemented  in  code  and  improve  the  quality  of 
the  design  materials  used  as  input  to  the  coding  phase. 


6.2. 1.3  Quality  and  Stability  of  the  Code:  The  coding  process  almost  al¬ 
ways  involves  some  detailed  design  beyond  that  specified  in  the  design  mate¬ 
rials.  As  a  result,  new  faults  can  be  introduced  at  this  stage.  Figures 
cited  by  Finfer  (Reference  8)  indicate  that  coding  errors  not  detected  until 
the  testing  stage  can  be  from  2  to  5  times  as  costly  to  correct  as  those 
detected  during  coding  and  checkout.  Once  again,  early  detection  decreases 
cost. 

The  code  verification  activity  of  IV&V  is  specifically  designed  to  detect 
anomalies  in  preliminary  versions  of  code.  It  relies  upon  inspection  rather 
than  program  execution  to  ensure  agreement  between  the  design  and  code  and  to 
check  for  faults  in  logic,  data  definition,  data  usage,  interfacing,  and  so 
on.  By  detecting  such  problems  before  the  program  has  been  integrated  for 
testing,  I V&V  can  decrease  the  cost  of  error  correction. 


This  approach  is  substantiated  by  Shooman  and  Bolsky  (Reference  6).  In  a 
study  of  error  detection  methods,  they  found  that  a  large  percentage  of  faults 
can  be  found  by  code  inspection  alone  and  that  this  method  is  cheaper  by  a 
ratio  of  25  to  1  than  techniques  involving  machine  testing. 

6.2. 1.4  Development  Team  Objectives:  In  Reference  37,*  Weinberg  reports 
that  the  objectives  of  a  software  development  team  exert  a  significant  influ¬ 
ence  on  programmer  productivity.  In  his  experiments,  development  teams  were 
given  the  same  program  specification  but  were  assigned  different  criteria  for 
success,  namely,  speed  of  program  execution  for  one  team,  speed  of  program 
development  for  the  other.  The  results  showed  that  the  different  objectives 
produced  markedly  different  results. 

The  application  of  this  principle  to  IV&V  lies  in  the  program  office's  abil¬ 
ity  to  foster  an  attitude  of  cooperative  competition  between  the  developer 
and  the  I  V&V  agency.  If  the  developer  knows  that  late  deliveries,  incom¬ 
pleteness,  inconsistency,  and  incorrectness  are  going  to  be  reported  by  the 
IV&V  agency,  he  will  tend  to  be  more  careful  to  meet  deadlines  and  more 
thorough  and  careful  in  his  work,  resulting  in  ultimate  productivity  gains. 

*WeinDerg,  G.  M. ,  "The  Psychology  of  Improved  Programming  Performance,"  Data¬ 
mation,  Nov.  1972. 
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6.2. 1.5  Programming  Language,  Development  Practices,  and  Configuration  Man¬ 
agement  Procedures  Used:  The  programming  language  and  development  and  config- 
uration  management  practices  used  on  a  project  have  been  shown  to  have  a  sig¬ 
nificant  effect  on  programmer  productivity.  In  Reference  23,  Doty  states  that 
development  of  a  program  in  assembly  language  rather  than  in  higher  order 
language  can  decrease  programmer  productivity  by  75%  and  that  use  of  modern 
programming  practices  can  result  in  a  67%  increase  in  productivity  for  large 
programs . 

The  inspection  methods  used  by  I V&V  permit  effective  detection  of  development 
standard  violations.  These  may  include  questionable  use  of  assembly  lan¬ 
guage,  incorrect  or  ineffective  use  of  modern  programming  practices,  viola¬ 
tions  of  accepted  development  practices,  inadequate  configuration  control 
procedures,  and  so  on.  The  I  V&V  agency  can  alert  the  program  office  to  these 
problems  so  that  adjustments  can  be  made  and  the  full  productivity  potential 
of  these  methods  can  be  achieved. 

6. 2. 1.6  Error  Correction  Procedures,  False  Starts,  and  Defect  Removal:  In 
Reference  30,  Wolverton  cites  studies  indicating  thalt: 

•  One  of  the  major  factors  affecting  productivity  is  false  starts. 

•  Two-thirds  to  four-fifths  of  programmer  time  is  spent  in  elimi¬ 
nating  faults  that  were  introduced  earlier. 

Supporting  these  studies  are  the  findings  of  Miyamoto  (Reference  36)  which 
indicate  that  on  the  average,  16.56  days  were  needed  to  correct  each  fault  in 
a  software  system  under  study. 

I  V&V  can  have  a  major  impact  on  both  false  starts  and  defect  removal  time. 
False  starts  can  be  reduced  by  the  improvement  in  requirements  and  design  re¬ 
sulting  from  IV&V.  Defect  removal  can  be  decreased  through  a  combination  of 
three  factors: 

•  Improved  requirements  and  design  mean  that  fewer  defects  are 
introduced  into  the  code  in  the  first  place. 

•  Unlike  development  testing,  which  identifies  the  effects  of  a 
program  failure  but  not  the  program  fault  causing  it,  I V&V 
identifies  the  specific  problem  in  the  code,  thereby  reducing 
the  Hf ind-and-f ix"  cycle  of  defect  removal. 

•  Early  detection  decreases  the  time  and  effort  required  to  cor¬ 
rect  problems  because  faults  are  found  before  the  modules  have 
been  integrated  and  tested. 

6.2. 1.7  Schedule  Compliance  and  Progress  Monitoring:  in  Reference  24, 
Brooks  states  that  schedule  overruns  result  not  from  major  calamities,  but 
from  the  cumulative  effects  of  day-to-day  slippage.  To  prevent  such  prob¬ 
lems,  it  is  crucial  that  all  of  the  activities  and  milestones  planned  for 
each  development  phase  be  completed  in  that  phase  and  not  be  allowed  to  ex¬ 
tend  into  the  next.  Completion  of  activities  and  milestones,  however,  can  be 


difficult  to  assess.  Timely  deliveries  of  products  that  are  inaccurate  or 
incomplete  may  provide  the  program  office  with  a  false  picture  of  overall 
project  status.  The  schedule  impacts  of  such  deliveries  may  not  surface  un¬ 
til  late  in  the  development  process. 


The  IV&V  agency's  technical  evaluation  yields,  as  a  side  effect,  a  continual 
assessment  of  the  overall  quality  and  status  of  the  development  project. 
Late  deliveries,  inadequate  materials,  failure  of  promised  access  to  develop¬ 
ment  facilities,  and  other  signals  can  indicate  to  the  IV&V  agency  that  the 
development  may  not  be  on  schedule.  The  IV&V  agency  can  then  alert  the  pro¬ 
gram  office  to  the  potential  problem  so  that  corrective  measures  can  be  taken. 


6. 2. 1.8  Adequacy  of  Formal  Reviews  and  Audits:  Formal  reviews  and  audits 
are  the  decision  points  of  the  development  process.  They  provide  the  program 
office  with  the  opportunity  to  review  the  developer's  progress  and  to  approve 
or  disapprove  development  products  before  the  next  development  or  life  cycle 
phase  begins.  IV&V  can  make  a  significant  contribution  to  these  reviews  and 
audits  by  providing  the  program  office  with  an  independent  evaluation  of  the 
materials  being  reviewed  and  of  the  review  or  audit  itself.  This  evaluation 
can  help  the  program  office  to  determine  whether  the  development  is  proceed¬ 
ing  satisfactorily  or  changes  need  to  be  made.  Informed  decisions  on  these 
issues  can  prevent  costly  schedule  overruns. 


6.2.2  Negative  Effects  of  IV&V 


Of  the  125  cost/productivity  factors  identified  in  Table  9,  IV&V  has  a  poten¬ 
tially  negative  effect  on  9.  The  following  paragraphs  discuss  these  factors 
and  IV&V's  effect  on  them. 


6.2.2. 1  Secondary  Resource  Expenditures:  The  developer  is  required  to 
supply  the  IV&V  agency  with  copies  of  specifications,  specification  change 
oages,  machine-readable  source  code,  computer  listings,  trade  study  reports, 
and  so  on.  Generation  of  these  deliverables  requires  computer  time,  use  of 
document  reproduction  facilities,  paper,  and  other  resources.  These  require¬ 
ments  impose  overhead  costs  on  the  development  effort. 

6. 2. 2. 2  Documentation  Requirements:  In  order  to  perform  requirements  veri¬ 
fication  and  design  verification  in  parallel  with  the  requirement  and  design 
phases,  the  IV&V  agency  requires  preliminary  deliveries  of  requirement  and 
design  materials.  To  provide  these  materials,  the  developer  may  be  required 
to  deliver  documents  that  would  not  otherwise  be  put  into  deliverable  form. 
These  added  documentation  requirements ,  while  having  possible  long-range  pro¬ 
ductivity  benefits,  take  time  away  from  the  development  effort. 

6. 2. 2. 3  Need  To  Share  Computing  Facilities:  On  some  projects,  the  IV&V 
agency  conducts  some  or  all  of  its  testing  on  the  same  computer  facilities 
used  by  the  developer.  These  facilities  may  be  government  furnished  equipment 
provided  for  the  use  of  both  contractors,  a  system  test  bed,  the  operational 
site,  or,  in  rare  cases,  the  developer's  own  facilities.  In  some  cases,  the 
sharing  may  involve  support  software,  test  software,  and  test  equipment  as 
well . 
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The  requirement  to  share  these  facilities  with  the  IV&V  agency  can  make  com¬ 
puter  facilities  less  available  for  development  testing,  increase  turn-around 
time,  and  complicate  the  computer  scheduling  process.  The  scope  of  these 
effects  can  be  controlled,  but  they  can  decrease  development  productivity. 

6. 2.2.4  Classified  Security  Environment:  Providing  the  I V&V  agency  with 
development  materials  and  sharing  computer  facilities  with  I  V&V  analysts  be¬ 
come  more  complex  when  the  program  being  evaluated  is  classified.  Transfer 
of  materials  from  one  agency  to  the  other  involves  formal  security  procedures 
involving  not  only  the  development  and  IV&V  teams  but  security  personnel  as 
well.  Computer  scheduling  must  take  into  account  not  only  the  amount  of  time 
needed  by  each  agency,  but  the  types  of  jobs  that  can  be  executed  concur¬ 
rently.  A  classified  security  environment  always  affects  programmer  produc¬ 
tivity  to  some  extent;  the  presence  of  the  IV&V  agency  adds  one  further 
compl ication. 

6. 2. 2. 5  Error  Reporting  and  Correcting  Procedures:  "Error  reporting  and 
correcting  procedures11  was  the  one  cost/productivity  factor  considered  to  be 
affected  both  positively  and  negatively  by  IV&V.  The  positive  aspects  were 
addressed  in  Section  6. 2. 1.6.  On  the  negative  side  is  the  time  required  for 
anomaly  report  processing.  Each  report  must  be  studied  and  understood.  A 
decision  must  be  made  as  to  its  validity  and  as  to  the  cost  effectiveness  of 
various  alternatives  for  action.  These  recommendations  must  then  be  conveyed 
to  the  program  office,  a  process  that  may  involve  discussions  including  the 
IV&V  agency. 

If  the  anomaly  report  concerns  a  problem  that  would  have  surfaced  during  pro¬ 
gram  development  or  operation,  development  time  is  saved  by  having  it  pointed 
out  in  an  anomaly  report.  If  the  report  is  incorrect  or  out  of  the  scope  of 
the  development  project,  anomaly  report  processing  time  clearly  outweighs  any 
benefits  gained.  If  the  report  concerns  a  maintainability  problem  that  would 
not  arise  in  the  development  process,  its  processing  would  tend  to  decrease 
development  productivity  but  have  a  positive  effect  on  life  cycle  cost.  The 
balance  between  positive  and  negative  effects,  therefore,  is  complex.  Section 
6.3.7  examines  this  question  quantitatively,  using  results  from  the  IV&V 
projects. 

6. 2. 2. 6  Customer  Interface.  Walston  and  Felix  (Reference  27)  identify 
"customer  interface  complexity"  as  a  factor  in  programmer  productivity.  They 
report  a  42%  decrease  in  productivity  when  this  interface  is  more  complex 
than  normal.  Though  not  explained  in  the  reference,  it  is  assumed  that  this 
complexity  is  determined  by  such  factors  as  the  degree  of  customer  involve¬ 
ment  in  the  development  process  and  the  amount  of  time  spent  in  interfacing 
with  the  customer. 

IV&V  can  increase  the  complexity  of  the  customer  interface  by  giving  the 
customer  more  visibility  into  the  development  process.  The  developer  may 
be  required  to  report  on  the  status  of  anomaly  reports,  present  arguments 
regarding  the  validity  of  anomaly  reports,  respond  to  the  customer  regard¬ 
ing  IV&V  requests  for  deliverables,  respond  to  the  customer  regarding  IV&V 
comments  about  the  development  process,  arrange  for  time  on  shared  computer 
facilities,  etc-  In  the  long  run,  a  better  informed  customer  enhances 
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productivity;  in  the  short  run, 
the  development  process. 


however,  these  added  demands  take  time  from 


Interfaces  With  Other  Agencies:  Nanus  and  Farr  (Reference  251  statP 
that  the  number  at  agencies  with  which  the  developer  must  deal  and  thei^  level 

veXPTehr'e"“HW;th  ?Str  dev.el.°P"!ent  have  an  impact  on  programmer  productiv¬ 
ity.  The  need  to  interface  with  the  IVSV  agency  falls  within  this  category. 


The  interface  with  the  JV&V  agency  may  include  discussions  of  anomaly  report 
validity;  discussions  of  anomaly  resolution;  technical  interchanges  about 
deliverables,  equipment,  and  other  aspects  of  the  development  or  IV&V  efforts; 
and  so  on.  While  such  interfaces  are  generally  kept  to  a  minimum  to  ensure 
the  independence  in  outlook  of  the  IV&V  agency,  they  do  require  time  of  devel¬ 
opment  team  members. 


6. 2. 2. 8  Paperwork;  Frederick  Brooks  (Reference  24)  reports  that  50%  of  a 

programmer  s  time  may  be  devoted  to  activities  other  than  programming.  Ma¬ 
chine  down-time,  unrelated  assignments,  meetings,  sickness,  personal  time, 
training,  paperwork,  and  so  on  account  for  the  rest.  IV&V's  effects  on  meet¬ 
ing  time  were  discussed  in  the  last  two  paragraphs.  IV&V  may  also  increase 

the  paperwork  required  from  the  developer.  The  increased  paperwork  may  in¬ 
clude  written  responses  to  anomaly  reports,  reports  and  logs  concerning  anom¬ 
aly  resolution,  and  other  records  required  by  the  program  office.  All  of 
these  demands  take  time  away  from  development  tasks. 

6. 2. 2. 9  Support  Personnel  Required:  In  Reference  23,  Doty  reports  that 

each  10%  increase  in  support  personnel  relative  to  programmers  and  analysts 
results  in  a  25%  decrease  in  product i vity.  This  figure  is  based  on  an  ex¬ 
pected  mix  of  20%  support  personnel  to  80%  programmers/analysts.  If  require¬ 
ments  for  deliverables,  computer  operators,  and  other  services  imposed  by  the 
presence  of  an  IV&V  contractor  require  the  addition  of  support  personnel,  this 
factor  would  come  into  play  in  reducing  productivity. 

6.3  Project  Results 

Hot  all  of  the  hypotheses  set  forth  in  Section  6.2  could  be  evaluated  from  the 
data  collected.  The  data  that  were  available,  however,  together  with  results 
from  the  literature,  provided  answers  to  the  following  questions: 

•  How  did  IV&V  cost  compare  with  development  cost  on  the  projects 
surveyed? 

•  To  what  extent  did  IV&V  contribute  to: 

The  quality  and  stability  of  requirements? 

The  quality  and  stability  of  the  design? 

The  quality  and  stability  of  the  code? 

Compliance  with  development  standards  and  configuration 

management  procedures? 

t  What  was  the  cost  benefit  of  early  error  detection? 
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t  What  were  the  positive  and  negative  effects  of  anomaly  report 
processing  on  programmer  productivity? 

The  following  paragraphs  discuss  these  results. 

6.3.1  Comparison  of  I V&V  Cost  to  Development  Cost 

Figure  22  compares  I  V&V  cost  with  software  development  cost  and  with  total 
acquisition  cost.  Discounting  Project  5,  for  which  cost  figures  were  not 
available,  I  V&V  project  costs  averaged  25%  of  development  costs  and  20%  of 
total  acquisition  costs.  These  would  appear  to  be  good  rules  of  thumb  for 
I V&V  cost  estimation. 

6.3.2  The  Effect  of  I V&V  on  Requirement  Quality  and  Stability 

Of  the  I  V&V  projects  surveyed,  only  one--Project  3— performed  a  standard 
requirements  verification.  Projects  1,  2,  and  5  reported  a  number  of  re¬ 
quirement  anomalies  (118  for  Project  2),  but  the  phase  in  which  they  were 
reported  was  past  the  time  at  which  significant  cost/productivity  benefits 
would  be  realized.  Project  4  reported  most  of  its  findings  in  the  weekly 
walk-throughs  performed  for  the  development  project,  making  the  effects  of 
its  requirements  verification  unavailable  for  the  study. 

Project  3  reported  96  requirement  anomalies  in  the  pre-code  phases  of  the 
project,  that  is,  early  enough  to  make  a  difference  in  the  quality  and  sta¬ 
bility  of  the  requirements.  Sixty-seven  more  requirement  problems  were  re¬ 
ported  after  code  release. 

The  breakdown  of  the  96  requirement  anomalies  was  as  follows: 

•  Incorrect  requirements:  38 

•  Inconsistent  requirements:  14 

•  Incomplete  requirements:  28  v' 

•  Other  content  problems:  13 

•  Presentation,  standards  problems:  3 

The  effect  of  these  anomaly  reports  was  to  clarify  the  program  requirements, 
improve  the  quality  of  the  requirement  specifications  used  as  input  to  the 
design  process,  prevent  false  starts  resulting  from  inadequate  understanding 
of  the  problem  to  be  solved,  and  decrease  the  number  and  magnitude  of  re¬ 
quirement  changes  needed  during  subsequent  development  phases. 

6.3.3  The  Effect  of  I V&V  on  Design  Quality  and  Stability 

Of  the  I V&V  projects  surveyed,  only  Project  5  conducted  a  standard  design  ver¬ 
ification.  This  analysis  resulted  in  75  anomaly  reports,  falling  into  the 
following  categories: 

•  Requirement  compliance:  1 

•  Choice  of  algorithm,  mathematics:  12 

•  Sequence  of  operations:  9 

•  Data  definition:  22 
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rcent  of  Total  Acquisition  Co 


Figure  22a.  I V&V  Cost  as  a  Percentage  of  Development  Cost 


Figure  22b.  I V&V  Cost  as  a  Percentage  of  Total  Acquisition  Cost 
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•  Data  handling:  23 

•  Interfaces,  I/O:  8 

The  effect  of  these  anomaly  reports  was  to  improve  the  quality  of  the  design 
materials  used  for  coding,  allow  design  problems  to  be  corrected  before  they 
were  propagated  into  the  code,  decrease  the  number  and  magnitude  of  design 
changes  needed  once  coding  was  under  way,  and  decrease  the  time  spent  in  false 
starts  and  defect  removal  during  the  coding  and  testing  phases. 

6.3.4  The  Effect  of  I V&V  on  Code  Quality  and  Stability 

The  5  I  V&V  projects  reported  329  code  anomalies  before  program  testing  was 
under  way,  that  is,  early  enough  to  have  significant  cost/productivity  bene¬ 
fits.  The  number  of  such  anomalies  on  each  project  was  as  follows: 


Project  1: 

40 

Project  2: 

66 

Project  3: 

58 

Project  4: 

83 

Project  5: 

82 

These  anomalies  fell  into  the  following  categories: 


Project 


Category 

T 

T~ 

T~ 

4 

5 

ATT 

Requirement/design  compliance 

1 

- 

17 

6 

1 

25 

Choice  of  algorithm/mathematics 

17 

25 

5 

17 

15 

79 

Sequence  of  operations 

3 

3 

10 

3 

14 

33 

Data  definition 

3 

10 

1 

17 

5 

36 

Data  handling 

9 

17 

9 

22 

21 

78 

Timing,  interruptibility 

- 

1 

- 

- 

- 

1 

Interfaces,  I/O 

2 

5 

5 

10 

4 

26 

Other  content  problems 

3 

1 

9 

2 

13 

28 

Presentation,  standards  compliance 

2 

4 

2 

6 

9 

23 

The  effect  of  these  anomaly  reports  was  to  improve  the  quality  of  the  program 
submitted  for  testing,  allow  for  correction  of  coding  problems  before  inte¬ 
gration  of  modules  made  this  process  more  difficult,  decrease  the  number  and 
size  of  proyrajn^ehcfnges  needed  during  testing,  and  decrease  the  time  devoted 
to  defect  removal  during  the  testing  phase. 
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6.3.5  The  Effect  of  IV&V  on  Standards  Compliance 

A  total  of  55  anomaly  reports  on  the  5  projects  concerned  compliance  with  de¬ 
velopment  standards  and  configuration  management  (CM)  procedures.  The  break¬ 
down  of  these  reports  by  project  was  as  follows: 

Anomalies  Concerning 

Project  Standards  "CM 

2 
8 

31  3 

1 

9  1 

The  standards-compliance  anomaly  reports  were  concerned  with  violations  of 
specified  standards  and  violations  of  wel 1 -accepted  programming  practices. 
These  anomaly  reports  alerted  both  the  developer  and  the  program  office  to 
potential  problems  in  the  production  of  code  and  documentation  that  could 
have  resulted  in  development  and  maintenance  problems.  The  configuration 
management  anomaly  reports  concerned  incorrect  version  identification  of  spec¬ 
ification  change  pages.  These  reports  helped  to  prevent  the  dissemination  of 
different  document  versions  bearing  the  same  configuration  control  markings. 

6.3.6  The  Cost  Benefits  of  Early  Detection 

A  major  objective  of  the  I V&V  study  was  to  quantify  the  effects  of  IV&V  on 
development  cost/producti vity.  By  applying  cost/productivity  figures  found 
in  the  literature  to  IV&V  results  of  the  five  projects  surveyed,  it  was  pos¬ 
sible  to  obtain  an  estimate  of  the  cost  benefits  associated  with  early  detec¬ 
tion  of  anomalies. 

A  basic  question  that  had  to  be  addressed  before  any  analysis  could  take 
place  was  which  anomaly  reports  to  consider.  Possibilities  were: 

•  All  of  them 

•  Only  those  that  were  accepted  and  corrected 

•  Only  those  that  affected  reliability 

Consideration  of  this  question  led  to  the  following  conclusions: 

«  An  anomaly  report  has  a  cost/productivity  benefit  on  the  devel¬ 
opment  effort  if  it  reports  a  problem  that  would  have  surfaced 
later  in  the  development  or  in  operation  and  required  correc¬ 
tion  at  that  tune. 

•  The  problems  that  would  have  surfaced  later  in  the  development 
effort  or  in  operation  and  required  correction  at  that  time  were 
those  affecting  program  reliability. 


1 

2 

3 

4 
5. 
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•  Reports  concerning  maintainability  anomalies  result  in  life 
cycle  cost  savings  but  not  necessarily  in  development  cost  sav¬ 
ings,  the  subject  of  the  analysis. 

•  Invalid  reports  or  those  that  were  not  acted  on  would  not  have 
a  cost/productivity  benefit. 

To  make  the  analysis  meaningful,  therefore,  it  was  limited  to  those  anomaly 
reports  that  were  concerned  with  program  reliability,  accepted  as  valid  by 
the  program  office,  and  acted  on  by  the  developer 

Two  analyses  were  performed.  The  approach  taken  for  the  first  was  as  fol¬ 
lows: 

•  Select  the  anomalies  meeting  the  criteria  outlined  above. 

•  Separate  the  selected  anomalies  according  to  the  development 
phases  in  which  they  were  detected. 

•  Apply  the  error-correction  cost  figures  reported  by  Wolverton 
in  Reference  30: 

-  Correction  during  requirements  definition:  $195 
Correction  during  design:  $489 

-  Correction  during  coding  and  checkout:  $977 
Correction  during  test  and  integration:  $7136 

•  Assume  that  all  of  the  anomalies  would  have  been  detected  dur¬ 
ing  development  testing  if  IV&V  had  not  been  performed. 

•  Calculate  the  cost  savings  realized  by  detecting  anomalies  be¬ 
fore  the  testing  phase. 

Table  10  shows  the  results  of  this  analysis.  In  keeping  with  the  assumption 
that  all  anomalies  would  have  been  detected  during  development  testing,  the 
anomalies  detected  during  the  testing  phase  are  shown  to  have  no  cost  bene¬ 
fit.  Those  detected  earlier  are  shown  to  have  cost  benefits  increasing  with 
their  distance  from  the  testing  phase.  Total  savings  are  shown  in  the  right¬ 
most  column. 

The  most  striking  finding  is  that  even  assuming  that  all  anomalies  would  have 
Deen  detected  by  the  developer,  substantial  cost  benefits  can  be  demonstrated 
based  on  early  detection  alone.  Even  with  270  anomalies  assumed  to  have  no 
cost  benefit  at  all,  the  figures  show  an  average  savings  of  $601,613  per  pro¬ 
ject.  For  Project  4,  the  savings  exceed  the  cost  of  the  IV&V  project,  despite 
the  fact  that  most  requirement  and  design  anomalies  were  not  documented  in 
anomaly  reports,  so  are  not  shown  in  the  analysis.  Project  3  savings  come 
close  to  its  cost.  The  lessons  shown  by  the  table  are  clear: 

•  The  earlier  anomalies  are  detected,  the  greater  the  cost 
benefits. 
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Testing  $3,136 


4  I V&V  can  pay  for  itself  through  early  detection  of  errors. 

The  second  analysis  was  identical  to  the  first  except  that  instead  of  assum¬ 
ing  that  all  anomalies  would  have  been  detected  during  development  testing, 
a  scenario  was  adopted  in  which  10%  of  the  anomalies  went  undetected  into  the 
operational  environment  and  were  found  there.  Figures  published  by  Finfer 
(Reference  8)  indicate  that  the  cost  to  correct  such  problems  is  15  times  the 
cost  if  detected  during  coding  and  checkout.  Combining  this  finding  with 
Wolverton's  cost  figures  produces  an  average  of  $14,655  to  correct  an  error 
once  the  program  becomes  operational. 

Table  11  shows  the  results  of  this  analysis.  The  average  savings  are  $714,097 
per  project,  up  from  $601,613  in  the  first  analysis.  Savings  shown  for  Pro¬ 
jects  3  and  4  exceed  the  cost  of  the  I V&V  efforts.  Not  shown  is  the  fact  that 
if  even  one  of  the  errors  not  detected  until  operational  use  had  a  mission- 
threatening  impact,  the  cost  benefit  of  1V&V  detection  would  be  much  higher. 
If  one  of  these  errors  had  a  life-threatening  impact,  the  benefit  would  be 
incalculable. 

6.3.7  Cost/Productivity  Fffects  of  Anomaly  Report  Processing 

The  time  required  to  analyze  and  respond  to  anomaly  reports  is  the  most  con¬ 
spicuous  imposition  of  I  V&V  on  development  producti vity.  The  question  that 
arises  is  whether  there  are  sufficient  productivity  gains  associated  with 
each  report  to  offset  the  time  that  must  be  spent. 

To  answer  this  question,  the  anomaly  reports  for  which  resolution  was  known 
were  divided  into  four  groups: 

t  Reports  that  were  concerned  with  reliability,  declared  valid, 
and  acted  on 

•  Reports  that  were  declared  invalid  or  that  were  withdrawn  by 
the  1 V&V  agency 

•  Reports  that  wore  declared  valid  and  were  acted  on,  but  did  not 
concern  reliability 

•  Valid  anomaly  reports  that  were  not  acted  on 

The  following  paragraphs  explore  the  cost/productivity  impacts  of  each  type 
of  report. 

6.3. 7.1  Effects  of  Corrected  Reliability  Anomaly  Reports:  The  first  group 
of  anomaly  reports  were  those  that  were  concerned  with  program  reliability, 
judged  valid  by  the  program  office,  and  acted  on  by  the  developer.  These 
reports  concerned  anomalies  that  would  have  shown  up  either  during  program 
testing  or  during  operational  use,  that  is,  anomalies  that  the  prograonier 
would  have  had  to  d^al  with  eventually.  The  analysis  assumed  that  all  of  the 
anomalies  would  have  been  detected  during  program  testing  rather  than  the  more 
troublesome  and  expensive  case  of  their  showing  up  in  operational  use. 
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In  a  paper  on  error  remediation  expenditures  (Reference  35),  Herndon  and 
Keenan  present  the  following  equation: 

Error  remediation  cost  equals  the  sum  of  the  costs  of: 


•  Error  handling: 

-  Trouble  report  generation 
Management  analysis 
Resolution  form  generation 

-  Configuration  control  board  actions 

•  Error  analysis 

•  Error  correction 

•  Retesting 

They  give  as  averages  the  following  figures  for  the  hours  required  for  each  of 
these  activities: 


Trouble  report  generation 

Management  analysis 

Resolution  form  generation 

Configuration  control  board  action 

Error  analysis 

Error  correction 

Retesting 


.17  hours 
.17  hours 
.25  hours 
.67  hours 

0-12.0  (say  6)  hours 
0-24.0  (say  12)  hours 
1.94  hours 


Total 


21.2  hours 


Other  authors  present  considerably  higher  estimates.  Miyamoto,  for  example, 
reports  that  the  average  "find-and-fix"  cycle  for  program  faults  in  a  large 
system  was  16.56  days  (Reference  36).  Multiplying  this  figure  by  a  conserva¬ 
tive  6  hours  per  day  results  in  99.36  hours,  nearly  4.7  times  the  figure  given 
by  Herndon  and  Keenan. 


The  figures  reported  in  the  literature  appear  to  depend  upon  the  size  and 
complexity  of  the  software  being  developed.  For  the  analysis  of  IVSV  results, 
it  was  decided  to  use  both  sets  of  figures  quoted  above  in  order  to  represent 
the  range  of  values  reported  in  the  literature.  Incorporating  Miyamoto's 
result  into  the  Herndon-Keenan  model  produces  the  following  estimate  for  error 
remediation: 


Trouble  report  generation  .17  to  .8  hours 
Management  analysis  .17  to  .8  hours 
Resolution  form  generation  .25  to  1.18  hours 
Configuration  control  board  action  .67  to  3.15  hours 
Error  analysis  6.0  to  28.2  hours 
Error  correction  12.0  to  56.4  hours 
Retesting  1.94  to  9.12  hours 


Total 


21.40  to  99.36  hours 


✓ 
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Shooman  and  Bolsky  (Reference  6)  add  that  the  typical  amount  of  computer  time 
used  to  diagnose  a  problem  is  0-4  runs,  using  0-30  minutes.  They  cite  as  a 
mean  .61  runs  and  13.5  minutes  (.225  hours)  of  computer  time. 

When  a  program  fault  is  detected  by  IV&V  rather  than  by  the  developer  in  pro¬ 
gram  testing,  the  impact  on  remediation  expenditure  is  as  follows: 

•  Trouble  Report  Generation:  No  trouble  report  has  to  be  gen¬ 
erated  by  the  development  test  team,  resulting  in  a  savings  of 
.17  to  .8  hours  per  anomaly. 

•  Error  Analysis:  The  time  needed  for  error  analysis  is  signifi¬ 
cantly  reduced  because,  unlike  a  trouble  report,  which  generally 
describes  only  the  symptoms  of  a  fault  as  they  were  observed 
during  testing,  an  anomaly  report  identifies  the  specific  error 
made  and  often  recommends  corrective  action.  A  conservative 
estimate  would  be  that  error  analysis  time  is  cut  in  half  when 
starting  with  an  anomaly  report  rather  than  a  trouble  report— 
a  savings  of  3.0  to  14.1  hours  per  anomaly. 

•  CPU  Diagnostic  Time:  The  computer  time  needed  for  diagnostic 

runs  is  also  reduced  because  of  the  specific  information  pro¬ 
vided  in  the  anomaly  report.  Assuming  a  reduction  by  half  re¬ 
sults  in  a  savings  of  .1125  computer  hours  per  anomaly. 

t  Paperwork:  Paperwork,  in  the  form  of  anomaly  response  forms, 
special  resolution  forms,  and  so  on,  may  or  may  not  be  greater 
than  that  required  in  response  to  trouble  reports.  Assuming 
that  the  time  required  for  paperwork  is  increased  by  half 
results  in  an  extra  .125  to  .55  hours  per  anomaly. 

•  Management  Analysis:  The  time  required  to  evaluate  tradeoffs 

in  correcting  the  problem  may  be  greater  in  responding  to  anom¬ 
aly  reports  than  trouble  reports  because  of  IV&V  and  customer 
concurrence  requirements.  Assuming  that  the  time  is  increased 
by  half  results  in  an  extra  .085  to  .4  hours  per  anomaly. 

•  Other  Factors:  All  other  factors  in  the  Herndon-Keenan  model 
would  be  the  same  whether  processing  an  anomaly  report  or  a 
developer-generated  trouble  report. 

Table  12  shows  the  results  of  this  analysis.  The  cost  figures  discussed  above 
have  been  multiplied  by  the  number  of  anomaly  reports  on  each  project  so  that 

the  total  hours  saved  and  the  extra  hours  expended  can  be  seen. 

Oespite  the  consistent  use  of  conservative  assumptions,  the  figures  indicate 
that  the  processing  cf  anomaly  reports  documenting  problems  that  would  have 
had  to  be  corrected  eventually  actually  saves  programmer  time.  The  estimated 
savings  for  each  anomaly  is  2.96  to  13.9  hours,  plus  6.75  minutes  of  computer 
time.  For  Project  4,  which  had  the  smallest  number  of  anomalies  in  this 
category,  estimated  savings  in  programmer  time  ranged  from  267  to  1,252  hours. 
For  Project  2,  with  the  greatest  number  of  anomalies,  estimated  savings  ranged 
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Table  12.  IV&V  Effects  on  the  Cost  of  Defect  Removal 
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from  640  to  3,006  programmer  hours.  It  should  be  noted  that  these  findings  do 
not  take  into  account  the  additional  cost  advantages  of  early  detection.  They 
assume,  in  effect,  that  all  anomaly  reports  were  submitted  in  the  testing 
phase. 

6. 3. 7. 2  Effects  of  Invalid  Anomaly  Reports:  One  way  in  which  IV&V  can  waste 
programmer  time  is  by  submitting  invalid  anomaly  reports.  Reports  may  be 
invalid  because: 

t  They  identify  faults  where  none  exist. 

•  They  recommend  changes  that  are  beyond  the  scope  of  the  devel¬ 
opment  effort. 

•  They  recommend  changes  that  jire  not  necessary  for  satisfactory 
program  performance. 

The  number  and  percentage  of  anomaly  reports  declared  invalid  on  the  five 
projects  was  as  follows; 


Project  1: 

9 

( 

Project  2: 

40 

(12*) 

Project  3; 

25 

(  5*) 

Project  4; 

3 

2S) 

Project  5: 

18 

list 

Total 

95 

( m 

An  average  of  6$  of  the  anomalies  reported  were  declared  invalid.  Project  2, 
with  12$,  had  the  highest  number.  When  questioned  about  this  figure,  project 
participants  explained  that  there  was  disagreement  between  the  developer  and 
the  IV&V  team  as  to  the  degree  of  accuracy  that  could  and  should  be  achieved 
in  program  calculations,  and  the  disagreement  resulted  in  quite  a  few  IV&V 
recommendations  being  declared  invalid. 

To  quantify  the  effects  of  processing  invalid  anomaly  reports,  the  following 
assumptions  were  made; 

•  The  human  and  computer  time  spent  in  analyjring  and  rejecting  an 
invalid  report  may  be  as  great  as  the  time  spent  in  processing 
a  valid  one. 

•  The  time  spent  for  paperwork  and  management  analysis  may  also 
bo  as  great  for  an  invalid  report  as  for  a  valid  one. 

•  The  other  costs  assoeia-M  with  error  remediation  do  not  come 
into  play  in  processing  invalid  reports. 

These  assumptions  yield  the  following  estimates  for  time  expended  on  each  in¬ 
valid  anomaly  report; 
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•  Error  analysts:  3.0  to  14.1  hours 

•  Computer  time:  6.75  minutes 

•  Paperwork:  .375  to  1.77  hours 

•  Management  analysis:  .225  to  1.2  hours 

•  Total  programmer  time:  3.63  to  17.07  hours 

Table  13  shows  the  results  of  these  assumptions.  The  amount  of  time  spent 
processing  each  invalid  anomaly  report  was  estimated  to  be  from  3.6  hours  to 
17  hours.  Thus  on  Project  2,  with  40  such  anomaly  reports,  the  amount  of  time 
was  145  to  683  hours,  costing  $5,800  to  $27,320.  On  Project  4,  with  only  3 
such  reports,  the  amount  of  time  was  11  to  51  hours,  costing  $440  to  $2,040. 

\ 

Minimizing  these  figures  should  be  a  goal  of  all  I V&V  projects.  The  program 
office  can  support  this  effort  by  making  both  the  developer  and  the  I V&V  con¬ 
tractor  aware  of  the  scope  and  nature  of  anomaly  reports  that  will  be  consid¬ 
ered  valid. 

6. 3. 7. 3  Effects  of  Corrected  Non-Reliability  Anomalies:  Anomalies  that  have 
been  corrected  'but " do"' not  concern  rel i abi  1  i ty  are  pri mari ly  maintainability 
anomalies.  These  anomalies  are  concerned  with  extraneous  code,  violations  of 
development  standards,  incorrect  documentation,  and  so  on.  These  problems 
would  not  arise  in  program  testing  and  therefore  do  not  fit  into  the  error 
remediation  analysis  presented  above. 

The  number  and  percentage  of  such  anomalies  was  as  follows: 


Project  1: 

13  ! 

:  5%) 

Project  2: 

18  l 

6% 

Project  3: 

252  | 

49% 

Project  4: 

64  I 

37%) 

Project  5: 

31  | 

All  378  (24%) 


Nearly  half  of  the  anomalies  on  Project  3  and  over  a  third  of  those  on  Project 
4  fell  into  this  category.  Percentages  for  the  other  projects  were  10%  or 
less. 

From  the  point  of  view  of  the  developer,  these  anomaly  reports  may  be  consid¬ 
ered  pure  overhead— they  must  be  dealt  with  even  though  they  do  not  help  to 
remove  operational  problems  that  could  impair  developer  testing.  The  cost 
benefits  of  these  anomalies  come  into  play  only  when  the  entire  life  cycle  of 
the  software  is  considered.  Correcting  them  decreases  the  productivity  of  the 
development  team,  but  Increases  the  productivity  of  the  maintenance  team.  No 
quantitative  data  on  this  tradeoff  could  be  found. 

6. 3. 7.4  Effects  of  Valid  Reports  That  Mere  Not  Acted  On:  Anomalies  in  the 
fourth  category  were  those  for  which  the  program  office  decided  that  it  was 
not  cost-effective  to  make  a  correction  even  though  the  problem  was  real.  The 
number  and  percentage  of  such  anomalies  was  as  follows: 


Table  13.  Cost  Effects  of  Invalid  Anomaly  Reports 


% 


Assumes  3  to  14.1  hours  per  anomaly  report 
Assumes  .1125  computer  hours  per  anomaly  report 
Assumes  .375  to  1.77  hours  per  anomaly  report 
Assumes  .255  to  1.2  hours  per  anomaly  report 
Assumes  $40  per  hour  manpower  cost 


Project  1: 

19 

(  8*) 

Project  2: 

19 

(  6%) 

Project  3: 

12 

(  2%) 

Project  4: 

12 

(  7%) 

Project  5: 

88 

(23%) 

All: 

150 

(10%) 

Project  5,  because  of  the  experimental  nature  of  the  development,  had  an  un¬ 
usually  high  number  of  this  type.  The  other  projects  show  more  typical 
results,  averaging  approximately  6%. 

The  cost  impact  of  these  anomaly  reports  could  not  be  determined.  On  the 
negative  side,  they  required  processing  time  without  producing  any  reliability 
or  maintainability  gains.  On  the  positive  side,  they  reported  real  problems 
that  may  have  been  encountered  in  testing,  and  therefore  may  have  saved  the 
time  that  would  have  been  devoted  to  those  problems.  No  way  was  found  to 
quantify  these  effects. 

6. 3.7. 5  Summary  of  Anomaly  Report  Processing  Effects;  Table  14  summarizes 
the  contents  of  Sections  6. 3. 7.1*4.  Tine  primary  conclusions  of  the  analysis 
were  as  follows: 

•  The  saving  of  programmer  time  on  the  valid,  fixed,  reliability 
anomalies  far  outweighs  the  time  expended  on  invalid  anomaly 
reports. 

•  The  time  spent  fixing  non-reliability  anomalies  results  in  long- 
range  savings  in  the  .form  of  more  maintainable  software. 

•  Valid,  unfixed  anomalies  have  the  mixed  effect  of  requiring 
handling  time  but  pointing  out  problems  that  the  developer  or 
maintainer  may  have  to  be  aware  of. 
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Table  14.  Summary  of  Anomaly  Report  Processing  Effects 


_ Anomaly  Report  Type _ 

Project  1: 

Valid,  fixed,  reliability 
1  rival  id 

Valid,  fixed,  non-reliability 
Valid,  not  fixed 

Project  2: 

Vai  id,  fixed,  rel  iabi  I  ity 
Invalid 

Valid,  fixed,  non-reliability 
Valid,  not  fixed 

Project  3: 


Number 


Cost/Productivity  Impact 


Savings  of  556-2,615  hours;  21.2  computer  hours 
Expenditure  of  32-154  hours;  1.0  computer  hours 
Development  time  expended;  maintenance  time  saved 
Indeterminate  effect 


Savings  of  640-3,006  hours;  24.3  computer  hours 
Expenditure  of  145-683  hours;  4.5  computer  hours 
Development  time  expended;  maintenance  time  saved 
Indeterminate  effect 


Valid,  fixed,  reliability 
Invalid 

Valid,  fixed,  non-reliability 
Val id,  not  fixed 

Project  4. 

Val id,  fixed,  r«l  iabi  1  ity 
In  val  id 

Valid,  fixed,  non-reliability 
Valid,  not  fixed 

Project  5: 


Valid,  fixed,  reliability 

115 

Invalid 

18 

Valid,  fixed,  non-reliability 

31 

Valid,  not  fixed 

m 

1  Pruetts: 

Valid,  fixed,  reliability 

?4? 

invul  i>! 

% 

Valid,  fixed,  non-reliability 

434 

Valid,  net  fixed 

159 

Savings  of  408-1,920  hours;  15.5  computer  hours 
Expenditure  of  90-42?  hours;  2.8  computer  hours 
Development  time  expended;  maintenance  time  saved 
Indeterminate  effect 


Savings  of  26M,252  hours;  10.1  computer  hours 
Expenditure  of  11-51  hours;  0.3  computer  hours 
Development  time  expended;  maintenance  time  saved 
Indeterminate  effect 


Savings  of  341-1,600  hours;  12,9  computer  hours 
Expenditure  of  66-303  hours,  2.0  computer  hours 
Development  time  expended;  maintenance  time  saved 
Indeterminate  effect 


Savings  of  2,212-19,391  hours;  P.4.0  computer  hours 
Expenditure  of  345-1,622  hours,  10,’  to--p4er 
Development  time  expended,  maintenance  ti*e  saved 
Indeterminate  effect 


7. 


CONCLUSIONS  AND  RECOMMENDATIONS 


The  IV&V  study  analyzed  the  results  of  five  IV&V  projects  to  determine  the 
effects  of  I V&V  on  software  reliability,  maintainability,  and  development 
cost/productivity.  The  following  paragraphs  present  the  study's  conclusions 
and  recommendations. 

7.1  Study  Conclusions 

Conclusions  resulting  from  the  study  were  as  follows: 

•  I  V&V  results  depend  significantly  upon  project  objectives  and 
directives— I  V&V  finds  the.  types  of  problems  it  is  directed  to 
find. 

•  The  primary  emphasis  of  I  V&V  is  on  software  reliability;  soft¬ 
ware  maintainability  is  deemphasized  on  many  projects. 

•  While  I V&V  *  s  effect  on  reliability  cannot  be  quantified,  the 
impact  appears  significant. 

Each  project  detected  an  average  of  150  anomalies  that  would 
have  affected  program  reliability  and  that  were  deemed  im¬ 
portant  enough  for  correction.  This  is  equivalent  to  2.2 
such  anomalies  per  thousand  machine  language  instructions. 

-  Thirteen  percent  of  these  anomalies  were  of  Hi severity, 
indicating  possible  threat  to  life  or  property;  another 
35%  were  of  Medium  severity,  indicating  serious  threat  to 
mission  objectives. 

None  of  the  programs  that  underwent  I  V&V  have  experienced 
operational  problems  that  required  modification. 

•  I V&V  is  being  underutilized  as  a  tool  for  improving  software 
maintainability. 

-  Software  maintenance  costs  are  approaching  75%  of  software 
life  cycle  cost. 

-  This  cost  can  be  reduced  by  designing  software  with  specific 
mai ntai nabi 1 i ty  character! sties. 

-  I V&V  is  ideally  suited  for  evaluating  these  characteristics. 

-  IV&V's  current  charter  regarding  maintainability  is  usually 
limited  to  evaluating  program  documentation  and  identifying 
latent  errors;  this  role  is  often  deemphasized. 

<*  On  one  project  where  maintainability  was  emphasized,  IV&V 
detected  330  anomalies  whose  correction  improved  software 
maintainability. 
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•  IV&V  is  cost-effective,  especially  if  applied  early. 

-  IV&V  costs  average  25%  of  development  cost  and  20%  of  total 
software  acquisition  cost. 

Of  125  factors  known  to  influence  programmer  productivity, 
IV&V  has  no  effect  at  all  on  90,  indicating  the  limited 
overhead  that  IV&V  places  on  the  development  process. 

Of  the  remaining  35  factors,  IV&V  has  a  positive  effect  on 
27,  a  negative  effect  on  9  (on  one  factor,  both  a  positive 
and  negative  effect  could  be  seen). 

IV&V  increases  programmer  productivity  by  saving  the  time 
that  would  have  been  devoted  to  false  starts  and  defect  re¬ 
moval  . 

IV&V  can  pay  for  itself  through  the  cost  benefits  provided 
by  early  detection  of  anomalies. 

7.2  Recommendations  for  IV&V  Planning  and  Management 

Study  results  led  to  the  following  recommendations  for  increasing  IV&V  effec¬ 
tiveness  on  future  projects: 

t  Concerning  reliability: 

Encourage  independence  of  IV&V  outlook  and  techniques  by 
controlling  the  degree  and  type  of  contact  between  the  de¬ 
veloper  and  the  IV&V  agency. 

-  Plan  the  IV&V  project  to  allow  for  early  detection  of  prob¬ 
lems  so  that  there  is  time  fur  adequate  redesign. 

-  Ensure  that  corrections  to  anomalies  are  reverified  by  the 
IV&V  agency. 

*  Concerning  maintainability: 

-  Direct  tne  IV&V  agency  to  evaluate  the  software  for  main¬ 
tainability  as  well  as  for  reliability. 

-  Draw  up  a  checklist  of  specific  maintainability  criteria 
such  as  those  in  Appendix  C. 

-  Schedule  the  development  and  IV&V  efforts  so  that  there  is 
time  after  the  conclusion  of  testing  for  the  IV&V  agency 
to  evaluate  final  program  documentation. 

-  Treat  maintainability  anomalies  as  seriously  as  reliability 
anomalies;  perhaps  establish  separate  criteria  for  High, 
Medium,  and  Low  severity  ratings  for  these  anomalies* 
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t  Concerning  cost/productivity: 

Stress  early  detection  of  anomalies. 

Include  in  the  IV&V  process  requirements  verification  and 
design  verification  performed  in  parallel  with  the  require¬ 
ment  and  design  development  phases. 

Ensure  delivery  of  preliminary  requirement  materials,  design 
materials,  and  code  so  that  IV&V  can  proceed  in  parallel 
with  the  development  and  provide  feedback  into  each  phase. 

Clarify  IV&V  scope  to  minimize  the  number  of  anomaly  reports 
declared  invalid. 

Minimize  the  overhead  associated  with  anomaly  report  han¬ 
dling  while  still  maintaining  high  visibility  into  anomaly 
report  resolution. 

7.3  Recommendations  for  Future  Study 

A  number  of  interesting  questions  arose  during  the  study  that  could  not  be 
answered  with  the  data  available.  The  following  paragraphs  identify  these 
questions  and  provide  some  insight  into  the  issues  involved  in  answering  them. 

7.3.1  IV&V  Productivity 

Considerable  attention  has  been  devoted  to  measuring  and  improving  programmer 
productivity.  To  our  knowledge,  however,  no  attempt  has  been  made  to  measure 
IV&V  productivity,  or  even  to  define  it. 

Programmer  productivity  is  measured  in  terms  of  amount  of  output  per  unit  of 
time,  for  example,  source  lines  delivered  per  month.  Attempts  to  transfer 
this  measurement  scheme  to  IV&V  fun  into  problems.  The  primary  output  of  an 
IV&V  team  is  anomaly  reports.  Can  an  analyst  be  required  to  find  so  many 
anomalies  per  week?  Is  the  analyst  who  detects  five  anomalies  in  one  week 
more  productive  than  the  analyst  who  detects  only  three,  or  in  fact,  the 
analyst  who  detects  none?  Are  10  Low-severity  anomalies  equivalent  to  1  High? 
If  20  anomalies  are  detected  in  March  and  10  in  April,  has  IV&V  productivity 
decreased  50X? 

The  answer  is  both  “yes"  and  “no."  Finding  anomalies  is  what  IV&V  is  about, 
but  so  is  ensuring  their  absence.  Thorough  analysis  of  an  error-free  subrou¬ 
tine  produces  no  anomaly  reports,  but  cannot  be  considered  a  nonproductive 
activity.  Its  output,  in  effect,  is  a  “stamp  of  approval"  for  the  subroutine, 
no  less  valid  a  product  of  IV&V  than  anomaly  reports. 

If  “anomaly  reports  per  unit  of  lime"  is  not  a  good  measure  of  IV&V  productiv¬ 
ity,  what  is?  It  would  seem  to  wake  more  sense  to  measure  IV&V  productivity 
in  terms  of  input  processed  than  output  produced:  lines  of  code  analyzed  per 
week,  pages  of  documentation  evaluated  per  month,  test  procedures  carried  out 
per  week,  and  so  on. 
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IV&V  productivity  measured  in  this  way  could  be  expected  to  be  influenced  by 
many  of  the  same  factors  that  affect  programmer  productivity.  Program  size, 
complexity,  application,  and  so  on  affect  the  difficulty  of  both  developing  a 
program  and  evaluating  it.  The  same  is  true  for  specification  quality,  pro¬ 
gramming  practices  used,  personnel  experience,  and  many  of  the  other  factors 
identified  in  Table  9.  One  factor  not  experienced  by  the  development  team  is 
the  IV&V  project's  dependence  upon  the  development  schedule.  When  the  de¬ 
velopment  schedule  slips  and  required  products  do  not  become  available,  the 
IV&V  schedule  necessarily  slips  as  well.  The  degree  of  cooperation  provided 
by  the  developer  can  also  affect  IV&V  productivity. 

Interesting  questions  concerning  IV&V  productivity,  then,  are: 

•  How  should  it  be  defined? 

•  How  do  factors  that  affect  programmer  productivity  affect  IV&V 
productivity? 

•  What  other  factors  affect  it? 

•  How  do  different  tools,  techniques,  and  procedures  affect  it? 

•  How  do  different  development  practices  affect  it? 

•  What  are  reasonable  productivity  rates  to  expect  on  a  project? 

•  How  can  IV&V  productivity  be  improved? 

7.3.2  Effects  of  Modern  Programming  Practices 

Modern  programming  practices  have  been  incorporated  only  recently  into  the 
types  of  software  verified  by  IV&V.  As  a  result,  the  IV&V  study  was  able  to 
include  only  one  such  project,  a  sample  clearly  too  small  to  permit  conclu¬ 
sions  to  be  drawn  about  the  effects  of  modern  programming  practices  on  IV&V. 
Questions  of  interest  are: 

•  What  types  of  anomalies  are  found  on  projects  using  modern  pro¬ 
gramming  practices? 

•  How  do  they  compare  with  the  results  of  other  projects? 

•  Are  different  IV&V  techniques  required  for  these  projects? 

•  Can  IV&V  help  to  evaluate  the  effectiveness  of  various  program¬ 
ming  practices? 

7.3.3  Effects  of  Program  Characteristics  on  IV&V  Results 

The  IV&V  study  had  limited  information  about  each  program  evaluated.  Of  con¬ 
siderable  interest  would  be  a  study  that  related  each  anomaly  to: 
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•  The  software  function  or  module  in  which  it  was  found 

•  The  specific  characteristics  of  that  module,  such  as  size,  com¬ 
plexity,  number  of  interfaces,  etc. 

»  The  development  and  testing  methods  used  on  the  development 
project 

•  The  development  standards  used  on  the  project 
t  Other  program  characteristics 

7.3.4  Effects  of  Various  I V&V  Tools  and  Techniques 

Software  tools  can  be  a  powerful  aid  to  IV&V  analysis  and  testing.  They  can 
detect  the  presence  of  certain  types  of  problems,  ensure  the  absence  of 
others,  and  aid  in  the  analysis  and  testing  activities.  An  interesting  sub¬ 
ject  for  study  would  be  the  detection  method  used  for  each  anomaly  reported. 
Of  particular  interest  would  be: 

•  The  relative  effectiveness  of  manual  versus  automated  analysis 

•  The  relative  effectiveness  of  specific  tools  and  techniques 

•  The  types  of  problems  detected  by  each  tool  and  technique 

•  The  types  of  tools  and  techniques  still  needed  by  IV&V 

7.4  Recommendations  for  Data  Collection 

IV&V  and  development  projects  generate  enormous  amounts  of  data,  not  all  of 
which  can  be  saved.  The  following  recommendations  are  for  IV&V  recordkeeping 
procedures  that  would  aid  in  future  studies  of  this  kind. 

t  Maintain  a  central  anomaly  report  log  for  each  program  eval¬ 
uated;  record  for  each  anomaly: 

-  Anomaly  report  number 

-  Anomaly  report  date  and  analyst 

-  Anomaly  location,  including  document  version,  program 
version,  routine  name,  if  applicable 

-  Anomaly  description 

-  Anomaly  category 

-  Special  circumstances  surrounding  the  anomaly 

-  Anomaly  effects 
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Anomaly  severity 
Development  phase  when  detected 
I V&V  phase  when  detected 

-  Method  or  tool  used  for  detection 
Anomaly  acceptance  or  rejection,  and  date 
Anomaly  resolution  and  date  of  resolution 
Materials  changed  and  nature  of  changes 
Acceptibility  of  resolution  to  the  IV&V  contractor 

•  Include  a  copy  of  each  anomaly  report  in  the  log  file. 

•  In  a  central  log  for  each  program,  record  for  each  routine: 

Language  used 
Size 

Function 

-  Number  of  interfaces 

-  1/0  characteristics 

•  In  a  project  management  log,  record: 

Monthly  man-loading  for  each  1VW  activity 
Identification  of  materials  evaluated 

-  Personnel  assignments  in  terms  of  pages  of  documentation  or 
lines  of  code  to  be  evaluated,  test  procedures  to  be  per¬ 
formed,  etc. 

-  Time  required  to  complete  each  assignment 

Number  of  anomalies  resulting  from  each  assignment 

-  Projoc*  costs 

-  Accurate  schedules  showing  development  deliverables,  IV&V 
activities,  1 V&V  deliverables 

-  Statement  of  objectives 

-  Criteria  for  assigning  severity  ratings 
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»*' 

I  List  of  tools  usad 

!  -  List  of  project  participants 

$  Information  that  could  be  recorded  by  the  developer  to  aid  in  the  analysis  of 
.IV&V  effects  would  include: 

•  Accurate  schedules 

§  Activities  that  must  be  performed  to  support  IV&V 

•  Man-hours  spent  performing  these  activities 

•  Cost  of  paper,  tapes,  CPU  time,  etc.,  for  deliverables  to  the 
IV&V  contractor 


•  Development  techniques  used 

•  Programmer  productivity  figures 

•  Test  results 

This  information  about  the  IV&V  and  development  efforts  would  be  an  invaluable 
contribution  to  future  studies  of  this  type. 
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APPENDIX  A 
PROJECT  SELECTION 


The  project  selection  activity  consisted  of: 

•  Establishing  selection  criteria 
t  Identifying  candidate  projects 

•  Selecting  the  projects  to  be  used 
The  following  paragraphs  describe  these  activities. 

Al.  SELECTION  CRITERIA 

The  selection  criteria  used  in  the  study  came  from  two  different  sources. 
Basic  criteria  were  set  forth  in  the  stub's  Statement  of  Work  (SOW).  These 
criteria  stated  that  there  were  to  be  at  least  five  I V&V  projects,  that  each 
was  to  involve  an  unclassified  Department  of  Defense  (DoOl  program,  preferably 
a  conroarid,  control,  communication,  and  intelligence  (C 3 1 )  application,  and 
that  each  was  to  involve  a  program  having  at  least  100,000  tines  of  source 
code. 

To  these  basic  requ iren«nts  were  added  additional  criteria  aimed  at  ensuring: 

•  A  balance  of  higher  order  language  { HOL )  and  assembly  language 
programs 

•  A  balance  of  real-time  and  nomreal-time  prograpis 

r  A  balance  of  nodem  software  engineering  and  traditional  devel¬ 
opment  practices 

t  A  balance  of  initial  development  and  modification  efforts 

•  MV  projects  performed  in  parallel  with,  rather  than  subsequent 
to,  the  development  effort 

t  Complete,  rather  than  on-going,  MV  projects 

•  MV  projects  that  Had  kept  good  records  N 

Table  A-l  suewarizes  the  two  sets  of  selection  criteria.  *** 

A2.  CANDIDATE  PROJECTS 

Thirty-five  projects  were  considered  as  candidates  for  the  study.  Table  A-2 
summarizes  their  characteristics. 


fHf  BilHQ  AuOK  M  »fT-  HIT  FILMS) 
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I V&V  Data  Availability  Gcod/Fair/Poor/ Incomplete  Good 


Table  A-2.  Information  About  Candidate  Projects 
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90 K  Ha  HOL/^srs  Both  Yes  Both  Yes  Complete  Good  Good  candidate 
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All  35  projects  were  performed  for  the  DoD.  Projects  1  through  4  involved 
C  I  applications;  the  others  did  not.  The  number  of  source  lines  ranged 
from  a  low  of  IK  to  a  high  of  35QK,  with  most  single  programs  in  the  range  of 
10K  to  40K  lines.  Approximately  a  third  of  the  projects  involved  some  classi¬ 
fied  components. 

Eleven  of  the  projects  dealt  with  programs  written  entirely  in  assembly 
language.  Four  dealt  with  programs  written  entirely  in  higher  order  language. 
The  remaining  20  involved  programs  or  systems  using  both  types  of  languages. 

Twenty-nine  of  the  projects  dealt  with  real-time  programs.  Eight  involved 
programs  developed  with  modern  software  engineering  practices.  Approximately 
half  involved  initial  development;  the  other  half  dealt  with  modifications  to 
existing  systems. 

Twenty-eight  of  the  IV&V  projects  were  performed  in  parallel  with  the  develop¬ 
ment  effort,  only  seven  were  not.  All  but  six  of  the  projects  were  already 
completed.  Sixteen  of  the  projects  were  considered  to  have  good  data  avail¬ 
ability,  seven  fair,  six  poor;  in  six  cases,  data  were  incomplete  because  the 
projects  were  not  yet  complete. 

A3,  SELECTION  PROCESS 

Of  the  35  candidate  projects,  12  met  the  100,000-line  criterion.  Since  five 
of  these  were  either  nonstandard  IV&V  efforts  or  were  in  too  early  a  stage  for 
inclusion  in  the  study,  the  project  selection  task  consisted  of  picking  five 
from  among  the  seven  eligible  projects:  3,  4,  12,  13,  14,  19  and  23. 

All  of  these  candidates  fulfilled  certain  of  the  criteria.  All  were  DoD 
projects.  All  provided  a  balance  of  HOL  and  assembly  language.  Any  subset 
would  provide  a  balance  of  initial  development  and  modification.  All  were 
performed  in  parallel  with  the  development  effort;  all  were  complete  except 
for  Project  23,  which  was  nearly  so;  and  all  were  considered  to  have  good 
availability  of  data. 

One  important  consideration  was  the  relationship  of  the  projects  to  one 
another.  Projects  3  and  4  Involved  two  versions  of  the  same  system,  as  did 
Projects  12,  13,  and  14.  The  seven  projects  taken  together,  therefore, 
represented  only  four  different  systems,  and  a  selection  that  included  at 
least  one  from  each  system  was  desirable.  Other  considerations  were  that 
Project  19  was  the  only  one  involving  modern  programming  practices  and  that 
Projects  3  and  4  were  the  only  ones  involving  the  desired  C3I  application. 

The  final  decision  was  to  use  Projects  3,  4,  12,  19  and  23.  Projects  3  and  4 
were  both  included  because  of  the  desirability  of  using  C31  projects  in  the 
study.  Project  12  was  selected  from  among  Projects  12,  13  and  14  because  it 
represented  the  initial  development  of  the  system.  Projects  19  and  23  rounded 
out  the  selection  by  bringing  into  the  study  the  other  two  candidate  systems. 
These  five  projects,  in  the  order  discussed,  were  renaned  Projects  1  through  5 
for  the  remainder  of  the  study. 
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APPENDIX  B 
DATA  COLLECTION 


The  data  collection  activity  consisted  of: 

§  Translating  the  study's  objectives  into  specific  questions  that 
could  be  answered  by  the  study 


•  Identifying  the  data  needed  to  answer  these  questions 

•  Developing  data  collection  forms 

•  Obtaining  project  records 

•  Recording  relevant  data 

•  Converting  the  data  to  machine-readable  form 
The  following  paragraphs  describe  these  activities. 


Bl.  KEY  QUESTIONS 


Table  B-l  identifies  the  key  questions  identified  for  the  study.  These 
questions  focus  on  the  number  and  characteristics  of  anomaly  reports  affecting 
software  reliability,  the  number  and  characteristics  of  anomaly  reports 
affecting  software  maintainability,  quantitative  data  concerning  IV&V's  effect 
on  development  cost  and  productivity,  and  ways  in  which  project  characteris¬ 
tics  affect  IV&V  results. 

B2.  REQUIRED  DATA 

Analysis  of  the  study's  key  questions  revealed  that  three  types  of  data  were 
needed: 

•  Data  concerning  each  anomaly  reported  by  1V4V 

•  Oata  concerning  IV&V  project  characteristics 

•  Data  concerning  development  project  characteristics 

Table  B-2  identifies  the  specific  information  identified  for  each  of  these 
catagories. 

B3.  DATA  COLLECTION  FORMS 

Three  data  collection  forms  were  developed  for  the  stu<ty,  corresponding  tc  the 
three  types  of  data  required: 

•  An  anomaly  questionnaire 
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Table  B-l.  Key  Questions  Identified  for  the  Study 


Questions  concerning  software  reliability: 

t  How  many  of  the  anomaly  reports  submitted  affected  software 
reliability? 

•  What  development  materials  did  they  involve? 

•  What  types  of  anomalies  were  they? 

•  How  severe  were  they? 

•  What  aspects  of  reliability  would  they  have  affected? 

•  What  was  the  operational  reliability  of  the  resulting  software? 
Questions  concerning  software  maintainability: 

•  How  many  of  the  anomaly  reports  submitted  had  a  direct  affect  on 
software  maintainability? 

•  What  development  materials  did  they  involve? 

•  What  types  of  problems  did  they  report? 

•  How  severe  were  they? 

•  What  aspects  of  maintainability  would  have  been  affected? 

•  What  were  the  indirect  effects  of  reliability  anomalies  on 
maintainability? 

Questions  concerning  development  cost  and  productivity: 

•  What  was  the  average  ratio  of  IV&V  cost  to  development  cost? 

•  What  factors  affected  this  ratio? 

•  In  what  ways  did  IV&V  Increase  development  cost? 

•  In  what  ways  did  IV&V  decrease  development  cost? 
t  What  was  the  cost  effect  of  early  detection? 

•  What  was  the  overall  cost  impact  of  IV&V? 

questions  concerning  the  improvement  of  IV&V  effectiveness; 

•  How  did  different  IV&V  project  characteristics  affect  IV&V 
results? 

•  How  did  different  development  project  characteristics  affet. 
IV&V  results? 


Table  B-2.  Data  Needed  From  Each  Project 


Data  concerning  each  anomaly  reported  by  IV&V: 

•  Location  (specification,  code,  etc.) 

•  Type  of  problem 

a  Probable  effects  if  left  uncorrected 

•  Severity 

•  Detection  date 

•  Detection  method 
t  Resolution 

•  Resolution  date 

Data  concerning  each  IV&V  project: 

•  Objectives 

•  Schedule 

•  Han*  loading 

•  Relationship  with  developer 

•  Tools  and  techniques  used 
e  Cost 

Data  concerning  each  development  project: 

•  Schedule 

•  Han-1 oading 

•  Development  practices  used 

•  Programmer  productivity 

•  Test  results 

•  Software  operational  performance 

•  Software  maintenance  requirements 

•  Cost 
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•  An  IV&V  project  questionnaire 

•  A  development  project  questionnaire 

Figures  B-l,  B-2,  and  B-3  illustrate  these  questionnaires. 

In  preparing  the  anomaly  questionnaire,  current  literature  concerning  error 
classification  was  surveyed.  Table  B-3  presents  a  sampling  of  the  error 
classification  schemes  found.  A  significant  characteristic  of  many  of  these 
schemes  was  their  focus  on  coding  errors.  Requirement  and  design  errors  were 
frequently  relegated  to  a  single  category  or  ignored  altogether.  Because  IV&V 
monitors  the  entire  development  process  and  reports  anomalies  in  requirement 
and  design  specifications  as  well  as  code,  these  schemes  were  unsuitable  for 
the  study. 

A  notable  exception  to  this  coding  orientation  was  the  classification  scheme 
devised  by  the  Software  Acceptance  Criteria  Panel  of  the  Joint  Logistics 
Commanders  Joint  Policy  Coordinating  Group  on  Computer  Resource  Management. 
This  scheme,  reported  by  Hartwick  (Reference  38)*  includes  categories  in  three 
areas:  specifications,  code,  and  data.  It  proved  the  most  useful  as  a  basis 
for  developing  anomaly  categories. 

B4.  OBTAINING  PROJECT  RECORDS 

Three  types  of  records  were  needed  for  the  study: 

•  IV&V  technical  results 

•  IV&V  project  data 

•  Development  project  data 

IV&V  technical  results  were  obtained  from  anomaly  reports  and  anomaly  resolu¬ 
tion  records  for  each  project.  Figures  B-4  and  B-5  provide  an  example  of  each 
of  these  forms.  IV&V  project  data  were  obtained  from  accounting  records, 
project  reports,  interviews  with  project  participants,  and  information  provid¬ 
ed  by  Air  Force  project  officers.  Development  project  data  were  obtained  by 
channeling  requests  through  RADC  to  the  appropriate  Air  Force  project  offi¬ 
cers.  Table  B-4  identifies  the  project  records  that  were  obtained. 

Bb.  RECORDING  THE  DATA 

The  data-reeording  activity  involved  completion  of  a  development  and  IV&V 
project  questionnaire  for  each  IV&V  effort  and  an  anomaly  questionnaire  for 
each  anomaly  reported.  The  development  questionnaires  were,  for  the  most 
part,  filled  out  by  the  Air  Force  project  officers  contacted  by  RADC.  The 


*  Hartwick,  R.  0*,  Software  Acceptance  Criteria  Panel  Report.  Joint  Logistics 
Commanders  Joint  Policy  Coordinating  Group  on  Computer  Resource  Management, 
Software  Workshop*  April  1979. 


Respondent 


ANOMALY  QUESTIONNAIRE 

1.  Program _ 

2.  Report  Date _ 

3.  Anomaly  Description 
a.  Anomaly  location: 


_Version _ 

Analysts 


(  )  System/ subsystem  specification 
(  )  Interface  specification 
(  }  Software  requirements  specification 
(  )  Before-code  design  specification 

b.  Brief  description  of  anomaly _ 


Report  No. 


(  )  Code 

{  )  After-code  design  specification 
(  )  User  documentation 
(  )  Other _ 


c.  Anomaly  category: 
Requirement  specification; 


Before-code  design  specification  or  code: 


Incorrect  requirements 
Inconsistent  requirements 
Incomplete  requirements 
Other  requirement  problems: 

(  )  Unclear,  untestable 
(  )  Unfeasible,  questionable 

M  Extraneous,  inappropriate 
Other 

Presentation  problems 
(  1  Standards,  development  practices 
(  )  Configuration  management 
{  )  Other 


After-code  design  specification  or 
user  documentation: 


Incorrect  documentation 
Inconsistent  documentation 
Incomplete  documentation 
Other  content  problems 
Presentation  problems 
)  Standards,  development  practices 
)  Configuration  management 
j  Other 


Requirement/design  compliance 
Choice  of  algorithm,  mathematics 
Sequence  of  operations 
Data  definition 
Data  handling: 

{  )  Initial iaation 
(  )  Addressing,  Indexing 
(  Misuse  of  flags 
j  )  Misuse  of  counters 
(  )  Shared  memory  locations 
(  )  Other 

Timing  or  incorruptibility 
Interfaces  or  1/0: 

(  }  Input  handling 
(  j  Output 

!  )  Hardware  interfaces 
(  j  External  software  interfaces 
(  )  Routine  interfaces 
Other  design/ code  problems 

H  Extraneous/inefficient 
Program  error  handling 
(  )  Other 

Design/code  presentation 

i)  Standards,  development  practices 
1  Configuration  management 
1  Comments,  annotations 
)  Other 


Other  development  problems  (  )  Comments,  annotatic 

I)  Hardware  system,  other  programs  (  )  Other 

j  Other  documents 
i  Unknown  origin 
1  Development  process 
}  Other 

d.  Special  circumstances: 

{  )  An  error  made  while  correcting  a  previous!) -reported  anomaiy 

I)  A  disagreement  among  development  materials  with  none  clearly  wrong 
)  A  non-optima),  rather  than  tncorreet,  development  decision 
)  A  latent  error— not  wrong  now  but  could  cause  maintenance  problems 
1  A  hold-over  from  a  previous  version  of  the  program 
)  A  ’clone'  of  a  previously-reported  anomaly 
(  )  Other . . 


Figure  B-l*  The  Anomaly  Questionnaire 
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ANOMALY  QUESTIONNAIRE— page  2 

4.  Consequences  of  Anomaly: 

a.  The  anomaly  could  affect  the  program's 

(  )  Development  (  )  Operation  (  )  Usability 

{  )  Verifiability  (  )  Maintainability  (  )  Other _ 

b.  If  it  could  affect  operation,  would  it  specifically  affect: 

(  )  Correctness  {  )  Security  (  )  Other _ 

(  )  Accuracy  precision  {  )  Efficiency 

c.  Severity  of  consequences: 

(  )  High  (  )  Medium  (  )  Low  (  )  Unknown 

5.  Detection  Information 

a.  IV&V  activity  at  time  of  detection: 

(  )  Requirements  verification  (  1  Validation/testing 

(  )  Design  verification  (  )  Documentation  verification 

(  )  Code  verification  (  )  Other _ 

b.  Development  phase  at  time  of  detection: 


(  1  Requirements  definition 

(  1  Testing 

(  )  Design 

(  )  Post-testing 

(  )  Coding  and  checkout 

(  )  Other 

c.  Tools  or  methods  that  resulted  in  detection: 
(  )  Manual  analysis,  specifically _ 


(  )  Program  execution,  specifically, 
(  )  Tool  use,  specifically _ 


6.  Anomaly  Resolution: 


a,  Anomaly  acceptance: 

)  Accepted  as  written 
}  Accepted  with  changes 
}  Rejected 

b.  Action  taken: 


\  Wi t hdrawn/ superseded 
)  Unknown 

)  Other _ 


(  ) 


Fixed  and: 

(  ) 

Fix/workaround  deferred  and: 

(  )  Fix  was  verified 

(  )  Taken  care  of  later 

i  j  Fix  found  wrong 

(  j  Not  taken  care  of  later 

{  )  Fix  not  verified 

(  )  Outcome  unknown  or  pending 

Workaround  adopted 

(  ) 

Action  unknown 

Negated  by  another  change 

(  ) 

Other  __ 

No  action  to  be  taken 


c.  Materials  changed  in  response  to  report: 


( 


System/ subsystem  specification 
Interface  specification 
Software  requirements  specification 
Design  specification 


(  )  Code 

(  )  User  doeuoente&ictt 
(  )  unknown 
(  )  Other _ 


d.  Resolution  date 


Figure  8-U  The  Anomaly  Questionnaire  (continued) 
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1V&V  PROJECT  QUESTIONNAIRE 

Respondent 

1. 

Program 

Version 

2. 

Cost 

of  IV&V  project: 

a. 

Total 

b. 

Labor 

c. 

Computer 

d. 

Documentation 

e. 

Other  support 

3.  Duration  of  project  {give  start  and  stop  dates): 

a.  Total _ 

b.  Requirements  verification _ _____ 


c.  Design  verification _ 

d.  Code  verification _ _____ 

e.  Testing _ 

f.  Documentation  verification 

4.  Man-months  expended: 

a.  Total 

b.  Requirements  verification 

c.  Design  verification _ _ 

d.  Code  verification 

e.  Testing 

f.  Documentation  verification 

5.  Relationship  with  developer  >  check  one: 
a.  Good 

b»  fair  —  some  hostility 

c.  Poor  —  very  poor  cooperation;  considerable  hostility 

6.  Tools  used  on  project: 


?.  Project  participants: 


Figure  3-2.  The  IV4V  Project  Questionnaire 
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DEVELOPMENT  PROJECT  QUESTIONNAIRE 

1.  Program _ 


Respondent- 
Version _ 


2.  Development  cost: 

a.  Total 


b.  Labor _ 

c.  Computer__ 


d.  Documentation- 

e.  Other  support- 


3.  Development  duration  (give  start  and  stop  dates): 

a.  Total _ _ _ 

b.  Requirements  definition  phase _ _ 

c.  Design  phase _ _ 

d.  Code  and  checkout  phase _ 

e.  Testing  phase _ _ 


4.  Man-months  expended: 


a. 

b. 
e. 


Total 


Requi rebuts  definition- 
lies  i  gri¬ 


ll.  Code  and  +  Heekout= 
e.  lost  ins 


S.  If  #agratat»er  productivity  figures  mere  kept,  please  provide  them, 
b.  Tools  used  on  project: 

a.  Kero  pro&leas/ercors  rs^aed  dumjy  program  testing? 

b.  If  so: 

•  He*  aanyj 

•  Please  provide  copies  of  problem/crror  reports  if  available. 

•  Please  provide  problom/error  resolution  mfomaticn  i*  available. 
?.  Operational  performance: 

a.  Were  prooleot/errors  detected  during  operational  wef¬ 
ts.  if  so: 

•  Kou  many? 

•  Please  provide  copies  of  probleajertor  reports  if  available. 

•  Please  provide  prvblew/errer  resolution  information  if  available. 


Figure  B-3.  The  Development  Project  Questionnaire 

-120- 


DEVELOPMENT  PROJECT  QUESTIONNAIRE  ~  page  i 
8.  Maintenance: 


When  did  the  program  go  operational? 

Has  the  program  been  modified  since  going  operational? 
If  so,  was  modification  due  to: 

_ User  requirements  change 

_ _ Problems/ errors  detected  in  operational  use 


d.  If  modification  was  due  to  problems/errors ,  describe  or  provide 
documentation  describing  the  needed  changes. 

9.  Which  of  these  programming  practices  were  used?  (See  RADC  Computer 
Software  Development  Specification  No.  CP  0787796100E) 

Yes  No  If  yes,  to  what  extent? 
If  no,  what  was  used 
instead? 

a.  Top  down  design  ■ — — — — 

•  At  program  level 

•  At  system  level 

b.  Program  support  library 

•  Hinual  P$l  _ _ _ 

e  Sasic  PSl  __ _ _ 

•  full  PSL  with  management 

data  col  lection  ami  reporting  _  _  ______ 

'.  Language  standards 

* 

•  Structured  code  accomplished 

with  simulated  constructs  __  __ 

e  Structured  code  accomplished 
with  preprocessor 

Wf*  vw  - - - -  —  ■ - p - , - 

e  Structured  code  directly 

compilable  _________ 

d.  coding  convent tons/pmeduees 

•  SASC-regui red  cooing 

conventions  ________ 

*.  '  NAB4  conventions  with  tode  ~~  ~ 

reading 

•  KADI  conventions  with  design  • 

code  reviews 

«•*  ---• - -  --  ■■■--  1 1-  -1  |  -  - 

*,  Personnel  crganiaatiun 

»  itsdlfitd  programmer  team  ^  __  . 

•  Full  programr  tea# 


Hgure  8-3.  The  Development  Project  Questionnaire  (continued) 


Table  B-3.  Error  Categories  Found  in  the  Literature 


Authors 


Major  Error  Categories 


Amory,  Clapp 
(Reference  39)* 


Rubey 

(Reference  2) 


Dana,  Blizzard 
(Reference  7) 


Thayer,  et  al. 
(Reference  S) 


0 


\ 


Input  data 
Internal  data 
Computation  procedures 
Control  procedures 
Interface  procedures 

Incomplete  or  erroneous  specification 
Intentional  deviation  from  specification 
Violation  of  programming  standard 
Erroneous  data  accessing 
Erroneous  decision  logic  or  sequencing 
Erroneous  arithmetic  computations 
Invalid  timing 

Improper  handling  of  interrupts 
Wrong  constants  and  data  values 
Inaccurate  documentation 

Incomplete  or  erroneous  specification 
Specification  violation  due  to  incorrect  implementation 
Violation  of  programming  practices 
Incorrect  data/instruction  access  and  storing 
Incorrect  logic  and  sequencing 
Incorrect  branching  and  jumping 
Incorrect  equation  computation  and  arithmetic 
Incorrect  timing  and  process  allocation 
Problems  with  interruptibility  and  data  coherency 
Incorrect  constant  value  and  data  formats 
Incorrect  documentation 
Erroneous  use  of  system  hardware/software 

Computation 

Logic  *  * 

Data  input 
Data  handling 
Data  output 
Interface 
Data  definition 
Data  base  *, 

Operation 
Ocher 

Documentation 


'Amory*  W. ,  and 
Software  Error 
Jan7l97b - - 


Clapp,  J*A.,  Engineering  of  Quality  Software  Systems 
Classification  Methodology),  RADC-iR-74-324 ,  vol.  V 


ft 


Table  B-3.  Error  Categories  Found  in  the  Literature  (continued) 


Authors _ 

Hartwick 
(Reference  38) 


Endres 

(Reference  40)* 


Major  Error  Categories 


Software  specification 

Unnecessary  functions 
Incomplete  requirements  or  design 
Inconsistent  requirements  or  design 
-  Untestable  requirements  or  design 
Requirements  not  traceable  to  higher 
specification 
Incorrect  algorithm 
Incomplete  or  inaccurate  interface 
specifications 


Code 

Syntax  errors 

Noncompliance  with  specifications 

-  Interface  errors 

-  Exception  handling  errors 

Shared  variable  accessing  errors 
Software  support  environment  errors 

-  Violation  of  programming  standards 

Operational  support  environment  errors 


Data 

-  Accuracy  • 
Precision 
Consistency 


Understanding  the  problem/choice  of  algorithm 

-  Machine  configuration  or  architecture 

-  Dynamic  behavior  and  communication 
between  processes 

-  Functions  offered 

-  Output  listings  and  formats 

-  Diagnostics 

-  Performance 


Implementation 

-  Initialization  of  fields  and  areas 

-  Addressability 

-  Reference  to  names 

-  Counting  and  calculating 

-  Masks  and  comparisons 


*  Endres,  A.,  “An  Analysis  of  Errors  and  Their  Causes," 
the  International  Conference  on  Reliable  Software,  April  19 


Proceedings  of 

rs\wriirm: 
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Table  B-3.  Error  Categories  Found  in  the  Literature  (continued) 

Authors _ _ Major  Error  Categories _ 

Estimation  of  range  limits 

Placing  of  instructions  within  a  module, 

bad  fixes 

Nonprogrammi ng  errors 

-  Spelling  errors  in  messages  and  commentaries 
Missing  commentaries  or  flowcharts 
Incompatible  status  of  macros  or  modules 
Other 


Bowen 

(Reference  41)* 


AN/SLQ-32 ( V ) 
v&v  SOW 
(Reference  41) 


Bowen 

(Reference  41) 


Design 
Interface 
Data  definition 
Logi  c 

Data  handling 
Computational 
Other 

Requirements 
Processing  design 
Data  base  design 
Interface  design 
Processing  construction 
Data  base  construction 
Interface  construction 
Verification 

Specification/documentation 

Expanded,  reduced,  or  erroneous  requirements 
Nonresponsive  program  design 

Incomplete  or  erroneous  program  design  specifications 
Erroneous  decision  logic  or  sequencing 
Improper  program  storage  or  response  time 
Improper  handling  of  interrupts 
Incorrect  module  or  routine  linkages 
Erroneous  arithmetic  computations 
Insufficient  accuracy  in  implementation  of  algorithm 
Inaccurate  or  incomplete  comments  in  prologue 
Erroneous  editing  for  new  version  update 
Incomplete  or  inconsistent  data  structure  definition 
Wrong  value  for  constant,  or  preset  data 
Improper  scaling  of  constant  or  preset  data 


*  Bowen,  J.  B.,  "Standard  Error  Classification  to  Support  Software  Reliabil¬ 
ity  Assessment. u  Proceedings  of  the  National  Computer  Conference,  1980, 
pp.  697-705.  ~ 


-124- 


if'"  »V'«vtV>'='5*'-“ 


Table  B-3.  Error  Categories  Found  in  the  Literature  (continued) 

Authors _  _ Major  Error  Categories _ 

Uncoordinated  use  of  data  by  more  than  one  user 

Erroneous  access  or  transfer  of  data 

Erroneous  reformatting  or  conversion  of  data 

Improper  masking  and  shifting  of  data 

Failure  to  initialize  flags,  counters,  data  areas 

New  error  introduced  during  correction 

Noncompliance  with  programming  standards  or  conventions 

Baker 

(Reference  42)*  Computational  errors 

Logic  errors 
Input/output  errors 
Data  handling  errors 

Operating  system/system  support  software  errors 
Configuration  errors 
Routine/routine  interface  errors 
Routine/system  software  interface  errors 
Tape  processing  interface  errors 
User  interface  errors 
Data  base  interface  errors 
User-requested  changes 
Preset  data  base  errors 
Global  variable/COMPOOL  definition  errors 
Recurrent  errors 
Documentation  errors 
Requirement  compliance  errors 
Unidentified  errors 
Operator  errors 
Questions 
Hardware  errors 

Design/requirement  logic  errors 

Logic  errors 
Data  handling  errors 
User-requested  changes 
Operator  errors 
Recurrent  errors 
Requirements  compliance  errors 
Computational  errors 


*  Baker.  W«  F.,  Software  Data  Collection  and  Analysis;  A  Real-Time  System 
Project  Hi  story  - 

t  Fries*  M.  J-.,  Software  Error  Data  Acquisition,  RADC-TR-77-130,  April 
1977. 


Fries 

(Reference  43 )T 
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Table  B-4.  Project  Records  Obtained  for  the  IV&V  Study 


s 

b 


$ 

V- 


i 


Needed  Project  Project  Project 

Information  1  2  3_ . 


Technical  Results 


Anomaly  Reports  x 

Anomaly  Resolution  Data  :  x 

IV&V  Project  Data 

Cost  x 

Schedule  x 

Man-loading 

Objectives  x 

Relationship  With 
Developer  x 

Tools  Used  x 

Participants  x 

Development  Project  Data 

Cost  x 

Schedule  x 

Man-loading  x 

Programmer  Productivity 
Test  Results  x 

Operational  Performance  x 

Maintenance  Requirements  x 

Programming  Practices  x 


Project 

4 


x 

x 


x 

X 

X 

X 

X 

X 

X 


X 

X 

X 

•a* 

X 

X 

N/A 

X 


Project 

5 


x 

x 


X 

X 

X 

X 

X 

X 


X 

X 

m 

X 

N/A 

N/A 

x 


w-rsi 


Figure  B-5.  Typical  Anomaly  Report  Form 
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IV&V  project  questionnaires  were  filled  out  using  Logicon  records  and  IVAV 
management  data  supplied  by  Air  Force  project  officers. 

Most  of  the  data-recordi ng  activity  was  devoted  to  filling  out  the  anomaly 
questionnaire.  This  process  consisted  of  the  following  steps  for  each  anom¬ 
aly: 

•  Recording  program  name,  version,  anomaly  report  number,  date, 
analyst,  location,  and  severity 

•  Writing  a  short  description  of  the  anomaly 

a  Making  judgments  as  to  anomaly  category  and  effects 

•  Correlating  the  anomaly  report  date  with  1V&V  and  development 
schedules  to  determine  the  phase  in  which  the  anomaly  was 
detected 

•  Cross-referencing  the  report  to  anomaly  resolution  forms  to 
determine  its  acceptance  and  resolution 

•  Determining,  often  by  inference,  the  detection  method  and 
special  circumstances  associated  with  the  anomaly 

The  following  paragraphs  discuss  lessons  learned  in  this  process. 

B5.1  Severity  Ratings 

The  study  originally  proposed  to  classify  anomalies  into  four  severity 
categories— Critical ,  Serious,  Moderate,  and  Trivial— as  had  been  done  in 
the  IVAV  study  conducted  by  Oana  and  Blizzard  (Reference  7).  All  five  of  the 
1V&V  projects  selected  for  the  study,  however,  used  only  three  levels— High, 
Medium,  and  Low— and  it  was  decided  to  adopt  these  three  levels  rather  than  to 
try  to  map  the  three  levels  given  on  the  anomaly  reports  to  the  four  levels 
proposed  for  the  study. 

Upon  closer  inspection,  it  turned  out  that  two  of  the  projects— Projects  1  and 
2— had  actually  used  six  severity  ratings:  High  with  nuclear  safety  implica¬ 
tions,  High  without  nuclear  safety  implications,  Medium  with  nuclear  safety 
implications,  and  so  on.  Discussions  with  project  participants  revealed  that 
simply  ignoring  the  nuclear  safety  implications  and  using  the  High,  Medium, 
and  Low  designations  would  be  inaccurate  because  anomalies  with  nuclear  safety 
implications  are  inherently  more  serious  than  those  without.  The  partici¬ 
pants'  recommendation  was  to  use  the  nonnuclear  safety  ratings  as  they  were, 
but  to  rate  High  and  Medium  nuclear  safety  anomalies  as  High,  and  Low  nuclear 
safety  anomalies  as  Medium.  This  is  the  approach  that  was  used. 

Another  interesting  discovery  was  that  there  were  two  distinct  approaches  to 
assigning  severity  ratings.  One  approach  asked  the  question:  "How  severe 
would  the  consequences  be  if  the  problem  made  possible  by  this  anomaly  were  to 
occur?"  The  other  asked  the  dual  question:  "How  likely  is  it  that  this 
anomaly  will  cause  an  operational  problem,  and  how  severe  would  the  conse¬ 
quences  bo  if  the  problem  occurred?" 
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The  two  approaches  can  result  in  considerably  different  severity  ratings. 
Many  anomalies  concern  a  remote,  yet  real,  set  of  circumstances  that  could 
affect  program  operation.  These  anomalies  would  be  rated  higher  using  the 
first  approach  than  the  second.  Anomalies  concerned  with  incorrect  documen¬ 
tation,  code  that  is  wrong  but  happens  to  work  correctly  in  the  current 

version,  and  other  such  problems  would  also  be  rated  differently  by  the  two 

approaches. 

The  question  that  arose  for  the  study  was  how  to  resolve  the  potentially 

inconsistent  ratings  for  the  five  projects.  Discussions  with  project  partic¬ 

ipants,  however,  led  to  the  conclusion  that  the  soundest  approach  was  to  use 
the  ratings  that  were  originally  assigned,  despite  their  different  interpre¬ 
tations.  Severity  ratings  are  by  nature  subjective,  and  if  the  1V&V  agency, 
DoD  project  officer,  and  developer  all  agreed  to  a  given  rating  at  the  time  of 
the  project,  that  rating  reflects  project  outcome  more  accurately  than  one 
that  might  be  imposed  later.  Except  for  the  nuclear  safety  anomalies,  there¬ 
fore,  all  severity  ratings  were  accepted  without  change. 

B5.2  When  is  a  "Typo"  not  a  "Typo11? 

In  categorizing  documentation  anomalies,  an  attempt  was  n&de  to  differentiate 
between  "substantive"  anomalies,  such  as  incorrect,  inconsistent,  and  incom¬ 
plete  facts,  and  "presentation"  anomalies,  such  as  format  errors  and  typo¬ 
graphical  errors.  The  distinction  turned  out  to  be  unclear  in  the  case  of 
typographical  errors. 

In  English  text,  most  typographical  errors  are  easily  recognizable  as  such. 
A  sentence  that  says  "The  program  shall  accept  400  inputs  per  sacond"  may  give 
the  reader  pause,  but  is,  after  a  moment's  thought,  understandable.  On  the 
other  hand,  if  the  sentence  says  "The  program  shall  accept  40  inputs  per 
second,"  where  "40“  should  be  "400,"  a  typographical  error  has  turned  into  a 
potential  development  disaster  if  not  caught  and  corrected. 

Because  of  the  wide  disparity  in  the  potential  effects  of  typographical 
errors,  each  such  anomaly  was  judged  on  it  own  merits.  General  guidelines 
that  emerged  were  that  errors  In  the  typing  of  numbers,  mathematical  symbols, 
variable  names,  and  set/use  table  entries  were  regarded  as  substantive  anom¬ 
alies;  those  for  which  Interpretation  of  the  intended  meaning  was  relatively 
clear  were  classed  as  presentation  problems. 

B5.3  Anomaly  Effects 

For  purposes  of  the  study,  the  effect  of  each  anomaly  was  at  least  as  impor¬ 
tant  as  its  cause.  During  the  course  of  the  data  collection  activity, 
certain  guidelines  emerged  for  determining  anomaly  effects.  The  following 
paragraphs  describe  these  guidelines. 

BS.3.1  Requirement  Specification  Anomalies 

The  impact  of  a  requirement  specification  anomaly  depends  upon  both  the  type 
of  problem  and  the  development  stage  in  which  it  is  detected.  Anomalies 
detected  before  the  appearance  of  design  or  code  were  generally  considered  to 
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affect  program  development,  verifiability,  operation,  and  maintainability. 
For  anomalies  detected  after  the  appearance  of  design  or  code,  the  following 
guidelines  applied: 

•  If  a  faulty  requirement  resulted  in  faulty  design  or  code, 
it  was  considered  to  have  an  impact  on  program  development, 
verifiability,  operation,  and  maintainability. 

t  If  the  design  and  code  were  correct  despite  the  faulty  require¬ 
ment,  the  requirement  anomaly  was  considered  to  have  an  impact 
on  verifiability  and  maintainability  or  on  maintainability 
alone,  depending  on  the  anomaly. 

B5.3.2  Design  Specification  Anomalies 

The  impact  of  an  anomaly  in  the  design  specification  was  also  determined  by 
both  the  type  of  anomaly  and  the  time  at  which  it  was  detected.  Anomalies 
detected  in  the  before-code  design  specification  were  regarded  as  design  anom¬ 
alies  and  were  usually  considered  to  have  an  impact  on  program  development, 
operation,  and  maintainability.  Usability  was  sometimes  affected;  verifiabil¬ 
ity  was  usually  not  since  software  is  tested  against  requirements  rather  than 
design.  Anomalies  detected  in  the  after-code  design  specification  were 
treated  as  follows: 

§  If  the  faulty  design  resulted  in  incorrect  code,  the  design 
specification  anomaly  was  considered  to  affect  development, 
operation,  and  maintainability. 

•  If  the  code  was  correct,  the  design  specification  anomaly 
was  considered  to  affect  maintainability  only. 

85.3.3  Code  Anomalies 

Most  code  anomalies  affected  program  operation.  Anomalies  affecting  usability 
Included  imp  lenient  at  ions  that  provided  unclear  output  messages,  or  cases  in 
which  input  formats  were  overly  restrictive.  Anomalies  affecting  naintaina- 
bility  included  cases  of  extraneous  or  inefficient  code,  incorrect  comments, 
inconsistent  Implementation,  and  code  that  was  incorrect  but  happened  to  work 
correctly  in  the  current  version. 

Bb.3.4  User  Documentation 

Most  user  documentation  anomalies  affected  program  usability.  A  few  were  con¬ 
sidered  to  affect  maintainability  as  well. 

85.4  Detection  Methods 


Of  considerable  interest  to  the  study  were  the  tools!  and  techniques  that 
resulted  in  the  detection  of  each  anomaly.  Unfortunately,  this  information 
was  not  generally  contained  in  the  anomaly  reports  and  wis  therefore  unavail¬ 
able  to  the  study.  In  many  cases,  the  IVAV  technique  could  be  inferred  from 
the  IV&V  activity  in  progress  at  the  tine  of  detection.  Requirements  verifi- 
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to  involve  program  execution.  For  code  verification,  however;  detection 
could  have  resulted  from  either  manual  analysis  or  use  of  static  analysis 
tools,  so  no  assumption  could  be  made. 

B.6.  GENERATING  THE  I V&V  DATA  TAPE 

The  final  task  of  the  data  collection  activity  was  preparing  a  magnetic 
tape  containing  the  data  collected.  This  task  consisted  of: 

•  Determining  the  required  tape  characteristics  from  the  Data  and 
Analysis  Center  for  Software  (DACS),  where  the  tape  was  to 
reside 

t  Designing  the  tape  format 

•  Selecting  an  encoding  scheme  for  the  data 

•  Encoding  the  data 

•  Transferring  the  encoded  data  to  magnetic  tape 

•  Generating  a  listing  of  the  tape  contents 

The  tape  and  listing  were  used  for  subsequent  data  analysis.  At  the  conclu¬ 
sion  of  the  study  they  we»e  delivered  to  RADC. 
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APPENDIX  C 


SOFTWARE  FEATURES  THAT  CONTRIBUTE  TO  MAINTAINABILITY 

Specific  software  features  that  contribute  to  maintainability  are  given  below 
(References  17,  19,  44*). 

Area  _ Feature _ 

Requirements  Correct  requirements 

Consistent  requirements 

Complete  requirements 

Testable  requirements 

Inclusion  of  traceability  information 

Design  Allowance  for  excess  computer  capacity 

Top-down  design 

Modular  design 

Modules  of  limited  size 

Single  function  for  each  module 

Separate  modules  for  input,  output,  computation 

Single  entry,  single  exit  for  all  modules,  except  for 
certain  computer  interrupts  and  error-condition  exits 

Initialization  and  housekeeping  functions  internal  to 
the  modules  needing  them 

Only  control  modules  able  to  make  abort  decisions 

Communication  between  modules  limited  to  defined  inter¬ 
faces 

All  control  data  passed  only  through  defined  interfaces 
Coherent  conceptual  organization 
Consistent  application  of  design  principles 

•Stanfield,  J.  R.,  and  Skrukrud,  A.  M.,  Software  Acquisition  Management 
Guidebook;  Software  Maintenance,  E$D-Tk-77-327 ,  Oct.  19/7 . 


-133- 


Area 


Design 

(continued) 


Code 


_ Feature _ _ 

Centralized  data  base 
Controlled  data  base 

Limited  access  to  data  base  by  each  module 
Procedures  to  define  and  control  data  base  entries 
Module  and  data  base  interfaces  not  overly  complex 
Data  base  designed  for  expansion  and  change 
Data  base  symbolically  defined 
Limited  equipment  interfaces 
Machine  dependencies  isolated  and  encapsulated 
Allowance  for  future  extensions 
Self-monitoring  features 

Full  implementation  of  design  features  that  contribute 
to  mainta inability 

Maintaining  a  reasonable  storage  and  time  margin 

Use  of  a  single  higher  order  language  if  oossible 

No  assembly  language  embedded  in  other  code  unless  ex- 
plicity  called  for 

Adherence  to  module  size  constraints 

Use  of  structured  programming 

Use  of  blank  cards  to  set  off  functional  blocks  vis¬ 
ually 

Use  of  indentation  to  reflect  block  structure 

Use  of  general  comments  preceding  each  module 

Use  of  adequate  comments  to  identify  flow  of  control  and 
purpose  of  each  section 

Adequate  commenting  of  time-sensitive  areas  to  alert 
oaintainers 


Area 


Code 

(continued) 


Documentation 


_  Feature _ - 

Use  of  symbolic  parameters  for  constants  and  basic  data 
structure  sizes 

Use  of  subroutine  arguments  rather  than  global  common 
Use  of  named  common 

No  sharing  of  variables  or  temporary  storage  locations 

No  self-modifying  code 

No  absolute  addressing 

No  relative  addressing 

No  embedded  constants  or  literals 

Symbolic,  meaningful  data  references 

No  code  that  implicitly  couples  one  module  to  another 

Avoidance  of  dynamic  allocation  of  resources 

Avoidance  of  unnecessarily  complex  arithmetic  ant  con¬ 
ditional  statements 

Avoidance  of  recursive/reentrant  coding 

Avoidance  of  unnecessari ly  complex  logical  structures 

Consistency  In  design  implementation,  1/0  processing, 
error  processing,  module  interfacing,  module/variable 
naming 

Inclusion  of  test  aids  or  implementation  to  support 
their  use 

Complete,  accurate  description  of  the  program  as  coded 

Incorporation  of  *11  design  changes  made  during  coding 
and  testing 

Time-sensitive  portions  of  code  clearly  identified  and 
described 

Nodular  organization 
Consistency  in  detail  and  style 
Emphasis  on  ease  of  use 
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Area 


Feature 


Documentation 

(continued) 


Configuration 

Management 


Inclusion  of  objectives  and  assumptions 

Avoidance  of  complexity 

Allowance  for  expandability  and  ease  of  change 

Accurate  status  records  for  software,  documentation, 
and  changes 

Accurate  date  and  version  indicators  in  source  listings 
and  documentation 


Adequate  control  and  documentation  of  changes  during 
devel opment 


Consistent  numbering  schemes  to  relate  corresponding 
source  listings,  documentation,  status  records 


Controlled  use  of  program  patches 
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MISSION 

of 

Rome  Air  Develojrment  Center 

RAVC  plan*  and  executes  research,  development,  test  and 
selected  acquisition  programs  In  support  of  Command,  Control 
Communication*  and  Intelligence  (C3I)  activities.  Technical 
and  engineering  support  loitkin  areas  of  technical  competence 
l*  provided  to  ESP  Program  Offices  (POs)  and  other  ESP 
element*.  The  principal  technical  mission  arcus  are 
communications,  electromagnetic  guidance  and  control,  sur¬ 
veillance  o f  ground  and  aerospace  objects.  Intelligence  data 
collection  and  handling,  information  system  technology, 
Ionospheric  propagation,  solid  state  sciences,  microuXLve 
physics  and  electronic  reliability,  maintainability  and 
compatibility. 


